2025-04-08 20:20:30.971 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:20:36.361 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:20:36.361 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:20:36.389 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:20:36.390 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:20:36.390 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:20:36.523 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:20:37.833 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:20:37.835 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:20:38.701 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT8.236172821S 2025-04-08 20:20:38.797 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:20:38.990 INFO 2996966 --- [main,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:20:38.990 INFO 2996966 --- [main,,] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:20:39.074 INFO 2996966 --- [main,,] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:20:39.149 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:20:39.149 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:20:39.149 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:20:39.149 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:20:39.149 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:20:39.150 INFO 2996966 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2025-04-08 20:20:39.157 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:20:44.519 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:20:44.520 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:20:45.978 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1438 ms. Found 173 JPA repository interfaces. 2025-04-08 20:20:45.986 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:20:45.986 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:20:46.179 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 192 ms. Found 16 JPA repository interfaces. 2025-04-08 20:20:46.179 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:20:46.179 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:20:46.230 INFO 2996966 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 51 ms. Found 5 JPA repository interfaces. 2025-04-08 20:20:49.122 INFO 2996966 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=535a2406-c3b5-3b8b-8149-b06173f75dfb 2025-04-08 20:20:49.900 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#550e7ef6' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.152 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#550e7ef6' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.193 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#399a5a52' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.206 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#77e1b84c' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.216 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#77e1b84c' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.244 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration' of type [org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.260 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metadbConfiguration' of type [com.oceanbase.ocp.server.common.config.MetadbConfiguration$$EnhancerBySpringCGLIB$$e71d31da] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.290 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration' of type [org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.299 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.305 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'loadBalancerClientsDefaultsMappingsProvider' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration$$Lambda$677/1688765722] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.344 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultsBindHandlerAdvisor' of type [org.springframework.cloud.commons.config.DefaultsBindHandlerAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.366 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpClusterProperty.MetaDbProperties' of type [com.oceanbase.ocp.core.config.OcpClusterProperty$MetaDbProperties$$EnhancerBySpringCGLIB$$7f617fe9] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.387 INFO 2996966 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:20:50.388 INFO 2996966 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2025-04-08 20:20:50.388 INFO 2996966 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:20:50.394 INFO 2996966 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2025-04-08 20:20:50.445 INFO 2996966 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2025-04-08 20:20:50.458 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:20:50.458 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:20:50.458 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaDataSource' of type [com.zaxxer.hikari.HikariDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.502 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.sql.init-org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties' of type [org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.527 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceScriptDatabaseInitializer' of type [org.springframework.boot.autoconfigure.sql.init.SqlDataSourceScriptDatabaseInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.554 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.566 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa.hibernate-org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.583 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.595 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hikariPoolDataSourceMetadataProvider' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration$$Lambda$445/405587614] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.651 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.713 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaVendorAdapter' of type [org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.737 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'entityManagerFactoryBuilder' of type [org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:20:50.920 INFO 2996966 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 20:20:51.232 INFO 2996966 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2025-04-08 20:20:51.985 INFO 2996966 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2025-04-08 20:20:54.909 INFO 2996966 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:20:56.144 WARN 2996966 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2025-04-08 20:20:56.999 WARN 2996966 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 20:20:56.999 WARN 2996966 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 20:21:00.358 INFO 2996966 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 20:21:00.425 INFO 2996966 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:21:00.442 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.459 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [com.sun.proxy.$Proxy294] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.587 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5418a659' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.844 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'propertyManager.MonitorDbExtraProperties' of type [com.oceanbase.ocp.core.property.PropertyManager$MonitorDbExtraProperties$$EnhancerBySpringCGLIB$$3c18037c] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.852 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'monitordbConfiguration' of type [com.oceanbase.ocp.server.common.config.MonitordbConfiguration$$EnhancerBySpringCGLIB$$24ec3671] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.926 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpAzoneModelConverter' of type [com.oceanbase.ocp.ha.maz.OcpAzoneModelConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.946 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5d767956' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.947 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5d767956' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.948 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#59489572' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.948 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27c173ea' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.949 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27c173ea' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.950 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#504b0a70' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:00.999 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.repository.config.JpaMetamodelMappingContextFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.007 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.mapping.JpaMetamodelMappingContext] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.921 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.934 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [com.sun.proxy.$Proxy300] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.994 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#29724ba7' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.994 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#29724ba7' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.995 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2a056ee6' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.995 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#158b5594' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.996 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#158b5594' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:01.997 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6293efd1' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.375 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.376 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [com.sun.proxy.$Proxy304] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.408 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2209622a' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.409 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2209622a' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.410 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ae8efc6' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.410 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c40e398' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.411 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c40e398' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.413 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#51d0a9cd' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.440 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.440 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [com.sun.proxy.$Proxy305] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.461 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d763ab2' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.461 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d763ab2' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.462 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#76171039' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.462 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7aadf751' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.462 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7aadf751' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.464 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4ce6e84f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.500 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.500 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [com.sun.proxy.$Proxy306] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.545 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5715f434' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.546 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5715f434' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.547 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6c0eeeaf' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.548 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5d2ab65' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.548 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5d2ab65' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.550 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#456ac723' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.577 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.578 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [com.sun.proxy.$Proxy307] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.646 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaTransactionManager' of type [org.springframework.orm.jpa.JpaTransactionManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.931 INFO 2996966 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action group type info: {"HOST_MANAGE":"HOST","ALL":"UNKNOWN","HOST_MAINTAIN":"HOST","ALL_READ":"UNKNOWN","OBPROXY_MAINTAIN":"OBPROXY","ORGANIZATION_READ":"ORGANIZATION","HOST_CREATE":"HOST","USER_MANAGE":"USER","ARBITRATION_READ":"ARBITRATION","ARBITRATION_CREATE":"ARBITRATION","BINLOG_MANAGE":"BINLOG","OBPROXY_MANAGE":"OBPROXY","HOST_READ":"HOST","BINLOG_READ":"BINLOG","AUDIT_READ":"AUDIT","USER_READ":"USER","ROLE_READ":"ROLE","CLUSTER_MANAGE":"CLUSTER","CLUSTER_CREATE":"CLUSTER","TENANT_MANAGE":"TENANT","CLUSTER_MAINTAIN":"CLUSTER","TENANT_MAINTAIN":"TENANT","CLUSTER_READ":"CLUSTER","BINLOG_MAINTAIN":"BINLOG","TENANT_READ":"TENANT","OBPROXY_READ":"OBPROXY","ARBITRATION_MAINTAIN":"ARBITRATION","ROLE_MANAGE":"ROLE","BINLOG_CREATE":"BINLOG","ARBITRATION_MANAGE":"ARBITRATION","OBPROXY_CREATE":"OBPROXY"} 2025-04-08 20:21:02.932 INFO 2996966 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action type info: {"ALARM:CREATE":"ALARM","SELFCURE:UPDATE":"SELFCURE","HOST:DELETE":"HOST","ROLE:CREATE":"ROLE","HOST:UPDATE":"HOST","CLUSTER:UPDATE":"CLUSTER","INSPECTION:UPDATE":"INSPECTION","CLUSTER:READ":"CLUSTER","PACKAGE:READ":"PACKAGE","ALARM:READ":"ALARM","TAG:DELETE":"TAG","BINLOG:CREATE":"BINLOG","ARBITRATION:CREATE":"ARBITRATION","SELFCURE:READ":"SELFCURE","*":"UNKNOWN","CLUSTER:DELETE":"CLUSTER","CREDENTIAL:DELETE":"CREDENTIAL","OBPROXY:READ":"OBPROXY","PROPERTY:READ":"PROPERTY","ARBITRATION:UPDATE":"ARBITRATION","OBPROXY:DELETE":"OBPROXY","TENANT:UPDATE":"TENANT","BINLOG:READ":"BINLOG","HOST:READ":"HOST","INSPECTION:CREATE":"INSPECTION","ALARM:UPDATE":"ALARM","PROPERTY:UPDATE":"PROPERTY","BINLOG:UPDATE":"BINLOG","PACKAGE:UPDATE":"PACKAGE","HOST:CREATE":"HOST","ROLE:DELETE":"ROLE","TAG:READ":"TAG","ROLE:READ":"ROLE","BINLOG:DELETE":"BINLOG","TASK:UPDATE":"TASK","USER:DELETE":"USER","CREDENTIAL:UPDATE":"CREDENTIAL","USER:UPDATE":"USER","TENANT:READ":"TENANT","TAG:UPDATE":"TAG","OBPROXY:CREATE":"OBPROXY","ARBITRATION:READ":"ARBITRATION","ROLE:UPDATE":"ROLE","ORGANIZATION:READ":"ORGANIZATION","TENANT:DELETE":"TENANT","USER:READ":"USER","AUDIT:READ":"AUDIT","TASK:READ":"TASK","TAG:CREATE":"TAG","INSPECTION:READ":"INSPECTION","ARBITRATION:DELETE":"ARBITRATION","USER:CREATE":"USER","PACKAGE:DELETE":"PACKAGE","ALARM:DELETE":"ALARM","CREDENTIAL:READ":"CREDENTIAL","OBPROXY:UPDATE":"OBPROXY","CREDENTIAL:CREATE":"CREDENTIAL","CLUSTER:CREATE":"CLUSTER"} 2025-04-08 20:21:02.932 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionMetaHelper' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ActionMetaHelper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.956 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultPerfConfiguration' of type [com.oceanbase.ocp.server.common.config.DefaultPerfConfiguration$$EnhancerBySpringCGLIB$$c440847f] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:02.991 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'authenticationFacadeImpl' of type [com.oceanbase.ocp.core.security.AuthenticationFacadeImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.016 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#785a8346' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.016 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#785a8346' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.017 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#47951aad' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.018 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1963270f' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.018 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1963270f' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.020 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#46d50b32' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.103 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.104 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [com.sun.proxy.$Proxy310] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.123 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOperator' of type [com.oceanbase.ocp.core.security.organization.operator.HostOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.143 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7a250d1f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.143 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7a250d1f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.144 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7c06717' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.144 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1175d60' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.144 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1175d60' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.146 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1ec1a7f2' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.210 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.211 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [com.sun.proxy.$Proxy312] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.228 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObClusterOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.252 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#52e54aaa' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.252 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#52e54aaa' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.253 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#63e8903e' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.253 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6b23bca2' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.253 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6b23bca2' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.254 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#44d58d15' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.318 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.319 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [com.sun.proxy.$Proxy313] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.339 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObproxyOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.355 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b308104' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.355 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b308104' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.356 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5d4f3b82' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.357 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7d081b40' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.357 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7d081b40' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.358 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#706b2ff0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.395 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.395 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [com.sun.proxy.$Proxy315] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.408 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObTenantOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.423 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4de6beac' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.423 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4de6beac' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.424 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#c211faa' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.425 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#17e96bfa' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.425 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#17e96bfa' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.426 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e557d9d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.481 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.481 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [com.sun.proxy.$Proxy316] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.504 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOperator' of type [com.oceanbase.ocp.core.security.organization.operator.UserOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.518 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#10ce529a' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.519 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#10ce529a' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.519 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#41b5ad5f' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.520 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75b7a38b' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.520 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75b7a38b' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.522 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7d78caff' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.546 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.547 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [com.sun.proxy.$Proxy318] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.565 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ArbitrationOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.580 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#38ddd7d0' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.580 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#38ddd7d0' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.581 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1ac1b109' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.582 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#41ad05d0' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.582 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#41ad05d0' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.583 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#103109d3' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.617 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.618 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [com.sun.proxy.$Proxy319] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.633 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogOperator' of type [com.oceanbase.ocp.core.security.organization.operator.BinlogOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.634 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceManager' of type [com.oceanbase.ocp.core.security.organization.ResourceManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.648 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3f8d7fb0' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.648 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3f8d7fb0' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.649 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#79c1233c' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.650 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7369aeec' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.650 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7369aeec' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.651 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5db504ce' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.683 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.684 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [com.sun.proxy.$Proxy320] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.716 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#368ea8a3' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.717 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#368ea8a3' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.718 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#a9313af' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.718 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#621b834' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.719 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#621b834' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.720 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#18e20fa8' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.739 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.740 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [com.sun.proxy.$Proxy322] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.773 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#24991011' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.774 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#24991011' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.775 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43e6bfb7' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.775 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e271caa' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.776 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e271caa' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.778 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#579c3641' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.805 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.805 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [com.sun.proxy.$Proxy323] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.827 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'getOrganizationService' of type [com.oceanbase.ocp.core.security.organization.OrganizationServiceImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.852 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75eb3fad' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.852 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75eb3fad' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.853 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1e7e5906' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.853 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#127b659c' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.853 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#127b659c' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.854 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7648167b' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.955 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.956 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [com.sun.proxy.$Proxy324] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.995 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27be341b' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.995 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27be341b' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.996 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7fc6a598' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.996 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#837ef79' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.996 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#837ef79' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:03.997 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#c3e363' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.026 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.027 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [com.sun.proxy.$Proxy325] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.053 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#58072e51' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.053 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#58072e51' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.054 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#71644fdf' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.054 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#79b82278' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.054 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#79b82278' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.055 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#30c338c8' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.117 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.118 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [com.sun.proxy.$Proxy326] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.155 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2603c822' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.155 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2603c822' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.156 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6add36' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.156 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b8acd9d' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.157 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b8acd9d' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.158 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#61ca95ab' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.188 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.188 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [com.sun.proxy.$Proxy327] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.216 INFO 2996966 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:21:04.319 INFO 2996966 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:21:04.319 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.338 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#776535b0' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.352 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#776535b0' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.353 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#425de255' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.353 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#725ceb38' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.354 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#725ceb38' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.355 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#14a33025' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.396 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.397 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [com.sun.proxy.$Proxy330] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.421 INFO 2996966 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:21:04.441 INFO 2996966 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:21:04.441 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obProxyQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObProxyQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.460 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#610480b5' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.461 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#610480b5' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.462 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1b621efe' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.462 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#76d84f07' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.463 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#76d84f07' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.464 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#37a6c6b0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.486 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.487 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [com.sun.proxy.$Proxy331] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.504 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4fda4c7e' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.505 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4fda4c7e' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.505 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#c4a1e8d' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.506 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4da3f4e7' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.506 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4da3f4e7' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.507 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7879f9ed' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.543 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.544 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [com.sun.proxy.$Proxy332] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.556 INFO 2996966 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:21:04.582 INFO 2996966 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:21:04.582 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogQueryCache' of type [com.oceanbase.ocp.core.cache.helper.BinlogQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.585 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceCache' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ResourceCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.587 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.599 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'adminResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.608 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'clusterResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.617 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.626 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.635 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.643 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'tenantResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.645 INFO 2996966 --- [main,,] c.o.o.c.s.a.r.ResourceSupplierManager : Resource supplier list: [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl@b62776b, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl@254a0ce8, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl@6a060305, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl@2d349056, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl@5c33ab81, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl@5780bed1, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl@45985fe9] 2025-04-08 20:21:04.645 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceSupplierManager' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.ResourceSupplierManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.671 INFO 2996966 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$f32d0107] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:21:04.685 INFO 2996966 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2025-04-08 20:21:05.128 INFO 2996966 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=192.168.1.61:8080, name=b88d8e1c7d 2025-04-08 20:21:05.212 INFO 2996966 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 192.168.1.61:8080 2025-04-08 20:21:06.045 INFO 2996966 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2025-04-08 20:21:06.075 INFO 2996966 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2025-04-08 20:21:06.075 INFO 2996966 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.98] 2025-04-08 20:21:06.245 INFO 2996966 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2025-04-08 20:21:06.245 INFO 2996966 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 27087 ms 2025-04-08 20:21:06.607 INFO 2996966 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2025-04-08 20:21:06.848 INFO 2996966 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:21:06.848 INFO 2996966 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2025-04-08 20:21:06.848 INFO 2996966 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:21:06.878 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:21:06.879 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:21:07.358 INFO 2996966 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2025-04-08 20:21:07.368 INFO 2996966 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : host 192.168.1.61, port 2881, database monitor_database 2025-04-08 20:21:07.369 INFO 2996966 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:21:07.369 INFO 2996966 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2025-04-08 20:21:07.369 INFO 2996966 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:21:07.369 INFO 2996966 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2025-04-08 20:21:07.374 INFO 2996966 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2025-04-08 20:21:07.374 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:21:07.374 INFO 2996966 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:21:08.019 INFO 2996966 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 20:21:10.279 INFO 2996966 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:21:10.660 INFO 2996966 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 20:21:10.661 INFO 2996966 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:21:24.887 INFO 2996966 --- [main,,] org.reflections.Reflections : Reflections took 5385 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 20:21:29.153 INFO 2996966 --- [main,,] org.reflections.Reflections : Reflections took 4195 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 20:21:33.406 INFO 2996966 --- [main,,] org.reflections.Reflections : Reflections took 4241 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 20:21:33.425 INFO 2996966 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2025-04-08 20:21:33.625 INFO 2996966 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2025-04-08 20:21:34.096 INFO 2996966 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2025-04-08 20:21:34.703 INFO 2996966 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2025-04-08 20:21:34.848 INFO 2996966 --- [main,,] c.o.o.a.service.TaskRepositoryService : create task success, task=TaskInfo(id=1, uuid=254bed2a-3c53-4ecf-b850-9c2422739b3a, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=WAITING, executorName=192.168.1.61:2996966, createAt=null, startAt=null, endAt=null, heartbeatAt=null, result=null) 2025-04-08 20:21:34.848 INFO 2996966 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : create task done, taskType=AlarmProcess, expectCount=1, createCount=1 2025-04-08 20:21:34.848 INFO 2996966 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2025-04-08 20:21:34.848 INFO 2996966 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2025-04-08 20:21:35.716 INFO 2996966 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=arbitration_group_auto_replace_failed,ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,arbitration_auto_replace_operation_info,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_merge_error,ob_tenant_name_mismatch, 2025-04-08 20:21:36.658 INFO 2996966 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table name=metric_data_daily, createTime=1744114820 2025-04-08 20:21:36.658 INFO 2996966 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Set table switch time, time=1744114820 2025-04-08 20:21:38.037 INFO 2996966 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:21:38.081 INFO 2996966 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:21:38.093 INFO 2996966 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2025-04-08 20:21:39.084 INFO 2996966 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:21:38.773+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion 2025-04-08 20:21:39.090 INFO 2996966 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:21:39.090 INFO 2996966 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:21:38.773+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion 2025-04-08 20:21:39.090 INFO 2996966 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:21:39.090 INFO 2996966 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 20:21:39.609 INFO 2996966 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/root/ocp-server/alarm 2025-04-08 20:21:39.783 INFO 2996966 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2025-04-08 20:21:39.833 INFO 2996966 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=192.168.1.61:2996966 2025-04-08 20:21:39.914 INFO 2996966 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 20:21:40.018 INFO 2996966 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 20:21:40.036 INFO 2996966 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2025-04-08 20:21:40.079 INFO 2996966 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2025-04-08 20:21:40.172 INFO 2996966 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Init consumer, consumerInfo=ShardingConsumerEntity(id=1, module=alarm-detect-by-target, name=192.168.1.61:8080, state=NEW, enabled=true, heartbeatTime=2025-04-08T20:21:40.160+08:00, createTime=null) 2025-04-08 20:21:40.181 INFO 2996966 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Init consumer, consumerInfo=ShardingConsumerEntity(id=2, module=alarm-detect-by-rule, name=192.168.1.61:8080, state=NEW, enabled=true, heartbeatTime=2025-04-08T20:21:40.173+08:00, createTime=null) 2025-04-08 20:21:40.195 INFO 2996966 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@259201bc 2025-04-08 20:21:40.195 INFO 2996966 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@4f3544e2 2025-04-08 20:21:40.199 INFO 2996966 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:21:40.199 INFO 2996966 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:21:40.377 INFO 2996966 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2025-04-08 20:21:40.380 INFO 2996966 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2025-04-08 20:21:40.747 INFO 2996966 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2025-04-08 20:21:41.333 INFO 2996966 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2025-04-08 20:21:41.357 INFO 2996966 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2025-04-08 20:21:41.623 INFO 2996966 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/root/software 2025-04-08 20:21:43.186 INFO 2996966 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:21:43.186 INFO 2996966 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:21:43.245 INFO 2996966 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:21:43.303 INFO 2996966 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:21:44.153 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2025-04-08 20:21:44.153 INFO 2996966 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2025-04-08 20:21:44.155 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2025-04-08 20:21:44.169 INFO 2996966 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2025-04-08 20:21:44.171 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2025-04-08 20:21:44.171 INFO 2996966 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2025-04-08 20:21:44.172 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2025-04-08 20:21:44.172 INFO 2996966 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2025-04-08 20:21:44.173 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2025-04-08 20:21:44.173 INFO 2996966 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2025-04-08 20:21:44.175 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2025-04-08 20:21:44.175 INFO 2996966 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2025-04-08 20:21:44.176 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2025-04-08 20:21:44.176 INFO 2996966 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2025-04-08 20:21:44.177 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2025-04-08 20:21:44.180 INFO 2996966 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2025-04-08 20:21:44.181 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2025-04-08 20:21:44.181 INFO 2996966 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2025-04-08 20:21:44.182 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2025-04-08 20:21:44.182 INFO 2996966 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2025-04-08 20:21:44.183 INFO 2996966 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2025-04-08 20:21:44.183 INFO 2996966 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2025-04-08 20:21:46.203 INFO 2996966 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:21:46.203 INFO 2996966 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 20:21:46.206 INFO 2996966 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:21:46.206 INFO 2996966 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 20:21:46.242 INFO 2996966 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=192.168.1.61:8080, consumerId=2, resources=[ShardingSlice(consumerId=2, type=DEFAULT, resourceIds=[18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 80, 81, 106, 107, 250, 251, 252, 253, 254])] 2025-04-08 20:21:50.475 INFO 2996966 --- [main,,] org.reflections.Reflections : Reflections took 4283 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 20:21:50.487 INFO 2996966 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2025-04-08 20:21:50.488 INFO 2996966 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2025-04-08 20:21:50.488 INFO 2996966 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2025-04-08 20:21:50.488 INFO 2996966 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2025-04-08 20:21:50.488 INFO 2996966 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2025-04-08 20:21:50.506 INFO 2996966 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on b88d8e1c7d 2025-04-08 20:21:51.528 INFO 2996966 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2025-04-08 20:21:52.140 INFO 2996966 --- [main,,] c.o.o.s.g.prefs.PrefsParameterService : Start loading prefs template. 2025-04-08 20:21:52.579 INFO 2996966 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Begin init binlog cluster parameter info. 2025-04-08 20:21:52.590 INFO 2996966 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Finish init binlog cluster parameter info, parameter blacklist: [] 2025-04-08 20:21:52.645 INFO 2996966 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Begin init binlog instance parameter info. 2025-04-08 20:21:52.653 INFO 2996966 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Finish init binlog instance parameter info, parameter blacklist: [] 2025-04-08 20:21:53.649 INFO 2996966 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2025-04-08 20:21:54.689 INFO 2996966 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2025-04-08 20:21:54.696 INFO 2996966 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=null, valid=false 2025-04-08 20:21:54.815 INFO 2996966 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Generate new rsa key, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 20:21:54.829 INFO 2996966 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.sensitive-data.encryption.public-key, previousValue=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB, updatedValue=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB, runningValue= 2025-04-08 20:21:54.835 INFO 2996966 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.sensitive-data.encryption.private-key, previousValue=MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALa0JzRq26/5U98SySQm2exhJk/WVdnRB21aFqDj+Q2EVhBW8spd6eej/xNCj3phEEjS3Y4l90mWdomi/ge45DRjua78WTIOzkdjH9swd9o4aacEZpFsTCrVmYgc7yWC3MKXmDq8yBtJECqO+bijfaqxFITG2qLIaajdCG4m0SsjAgMBAAECgYAZJhEjcPYOcfZuhUp0lHh4xiKc6LbmhZRNJBJrURYOG0Ixe5Zt7qFsxjEtOINBaCzyqTvf/i/czD5C0dG/xF588As7R4+/eMzBdf/Oy5SuiIJBzPo/NfTOzSdAdB/8OOcr1paYjm1mSXFHcR0c7cocwLYoAZvas/9j1HsOHBJW6QJBAOEPukYFSuu/xo0pUv53VVZNgRvW8VrahVf46At4M6IpnJtAvNk1SdVA9a51TtnjxQgZI0XyMsCyClkAN7p1UL0CQQDP0cot2ycOuzvwvpNzUK8wlOrC9WJePCHJ2ZAqbM3P6sbs64lwMFqUmTwEZmNKqgdoI+m5yCmoBfKwcoAsuNlfAkAjdCrP5P2MbcWFb0EXvQzAb60rCY0z1sCyvw8u31yi6xM0kdouDWr4ISeSuAAX6v2a7PrVhfXgzSs8IgClat2xAkAE2TcnVLqXjGS8nCWihhpgPskvXTw7SJndNLc5ViqDFE4pzuD4/hfO6bhNvfnJDyh3FIdqbzIRmnnsBe/Zm0JvAkEA2jeOFU6xK4OfrNDBw9K8hQAlFod6czK9xu4mveso6C2CWecY9ih8f5CutvCewVFhal/p65C2KyWXwz7uueXXhQ==, updatedValue=MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBALa0JzRq26/5U98SySQm2exhJk/WVdnRB21aFqDj+Q2EVhBW8spd6eej/xNCj3phEEjS3Y4l90mWdomi/ge45DRjua78WTIOzkdjH9swd9o4aacEZpFsTCrVmYgc7yWC3MKXmDq8yBtJECqO+bijfaqxFITG2qLIaajdCG4m0SsjAgMBAAECgYAZJhEjcPYOcfZuhUp0lHh4xiKc6LbmhZRNJBJrURYOG0Ixe5Zt7qFsxjEtOINBaCzyqTvf/i/czD5C0dG/xF588As7R4+/eMzBdf/Oy5SuiIJBzPo/NfTOzSdAdB/8OOcr1paYjm1mSXFHcR0c7cocwLYoAZvas/9j1HsOHBJW6QJBAOEPukYFSuu/xo0pUv53VVZNgRvW8VrahVf46At4M6IpnJtAvNk1SdVA9a51TtnjxQgZI0XyMsCyClkAN7p1UL0CQQDP0cot2ycOuzvwvpNzUK8wlOrC9WJePCHJ2ZAqbM3P6sbs64lwMFqUmTwEZmNKqgdoI+m5yCmoBfKwcoAsuNlfAkAjdCrP5P2MbcWFb0EXvQzAb60rCY0z1sCyvw8u31yi6xM0kdouDWr4ISeSuAAX6v2a7PrVhfXgzSs8IgClat2xAkAE2TcnVLqXjGS8nCWihhpgPskvXTw7SJndNLc5ViqDFE4pzuD4/hfO6bhNvfnJDyh3FIdqbzIRmnnsBe/Zm0JvAkEA2jeOFU6xK4OfrNDBw9K8hQAlFod6czK9xu4mveso6C2CWecY9ih8f5CutvCewVFhal/p65C2KyWXwz7uueXXhQ==, runningValue= 2025-04-08 20:21:54.835 INFO 2996966 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Refresh and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 20:21:54.949 INFO 2996966 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2025-04-08 20:21:55.232 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 20:21:55.264 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 20:21:55.285 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:21:55.294 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:21:56.521 INFO 2996966 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 192.168.1.61:8080 2025-04-08 20:21:56.530 INFO 2996966 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2025-04-08 20:21:56.568 WARN 2996966 --- [pool-13-thread-1,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 20:21:56.568 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 20:21:56.916 INFO 2996966 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2025-04-08 20:21:56.929 INFO 2996966 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2025-04-08 20:21:58.521 INFO 2996966 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/root/ocp/ocp-server/lib 2025-04-08 20:21:58.521 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2025-04-08 20:21:58.577 ERROR 2996966 --- [main,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.IllegalArgumentException occurred with code error.file.meta.not.found, and args [built-in, obproxyd.sh] 2025-04-08 20:21:58.577 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Script:obproxyd.sh not exist. 2025-04-08 20:21:58.577 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2025-04-08 20:21:58.662 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=1 2025-04-08 20:21:58.756 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=1, name=obproxyd.sh 2025-04-08 20:21:58.778 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 20:21:58.830 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm, objectId=2 2025-04-08 20:21:59.569 WARN 2996966 --- [pool-13-thread-1,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 20:21:59.569 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 20:22:01.428 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=2, name=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 20:22:01.517 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 20:22:01.526 INFO 2996966 --- [rpm-extract1,e3b09a862455e604,169233fa367bf000] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:22:01.526 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.545 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.548 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.557 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm, objectId=3 2025-04-08 20:22:01.558 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.566 INFO 2996966 --- [rpm-extract1,e3b09a862455e604,169233fa367bf000] c.o.o.service.event.RpmExtractListener : Current package is ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm, do not need to extract rpm. 2025-04-08 20:22:01.570 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.582 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.592 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.593 WARN 2996966 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:22:01.945 INFO 2996966 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Init consumer, consumerInfo=ShardingConsumerEntity(id=3, module=metric-collect, name=192.168.1.61:8080, state=NEW, enabled=true, heartbeatTime=2025-04-08T20:22:01.942+08:00, createTime=null) 2025-04-08 20:22:01.946 INFO 2996966 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@31f8a6b0 2025-04-08 20:22:01.947 INFO 2996966 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:22:02.569 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 gets the leadership... 2025-04-08 20:22:03.158 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=3, name=ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 20:22:03.161 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:22:03.167 INFO 2996966 --- [rpm-extract1,23cf10f6dc92c6a2,242376cac54b67c8] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:22:03.169 INFO 2996966 --- [rpm-extract1,23cf10f6dc92c6a2,242376cac54b67c8] c.o.o.service.event.RpmExtractListener : Current package is ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm, do not need to extract rpm. 2025-04-08 20:22:03.221 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm, objectId=4 2025-04-08 20:22:03.258 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=4, name=oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:22:03.263 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:22:03.267 INFO 2996966 --- [rpm-extract1,7c3dceab87bc7df1,70d21a8facd0dc60] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:22:03.270 INFO 2996966 --- [rpm-extract1,7c3dceab87bc7df1,70d21a8facd0dc60] c.o.o.service.event.RpmExtractListener : Current package is oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm, do not need to extract rpm. 2025-04-08 20:22:03.298 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm, objectId=5 2025-04-08 20:22:04.947 INFO 2996966 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:22:04.976 INFO 2996966 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:22:05.569 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:05.583 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:07.953 INFO 2996966 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:22:07.953 INFO 2996966 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2025-04-08 20:22:08.583 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:08.583 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:11.584 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:11.584 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:14.585 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:14.586 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:17.586 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:17.587 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:20.142 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=5, name=oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:22:20.146 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 20:22:20.160 INFO 2996966 --- [rpm-extract1,3f52ab6417b49547,5175273c916ca1df] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:22:20.162 INFO 2996966 --- [rpm-extract1,3f52ab6417b49547,5175273c916ca1df] c.o.o.service.event.RpmExtractListener : Current package is oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm, do not need to extract rpm. 2025-04-08 20:22:20.177 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=sys-package, name=obproxy-ce-4.3.3.0-5.el8.x86_64.rpm, objectId=6 2025-04-08 20:22:20.587 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:20.589 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:22.371 INFO 2996966 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=6, name=obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 20:22:22.378 INFO 2996966 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.5-20250319105844 2025-04-08 20:22:22.382 INFO 2996966 --- [rpm-extract1,9d5ded4e72432413,e37878bc9987b1dd] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:22:22.385 INFO 2996966 --- [rpm-extract1,9d5ded4e72432413,e37878bc9987b1dd] c.o.o.service.event.RpmExtractListener : Current package is obproxy-ce-4.3.3.0-5.el8.x86_64.rpm, do not need to extract rpm. 2025-04-08 20:22:22.394 INFO 2996966 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.5-20250319105844, updatedValue=4.3.5-20250319105844, runningValue= 2025-04-08 20:22:22.441 INFO 2996966 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"update_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition","timeoutNs":0},"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event","timeoutNs":0},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception","timeoutNs":0},"cleanStorageObjectBlock":{"batchIntervalMillis":10000,"batchSize":10,"condition":"file_id not in (select id from storage_object_meta)","priority":0,"tableName":"storage_object_block","timeoutNs":0}} 2025-04-08 20:22:22.885 INFO 2996966 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2025-04-08 20:22:23.130 INFO 2996966 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2025-04-08 20:22:23.132 INFO 2996966 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2025-04-08 20:22:23.589 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:23.590 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:24.432 INFO 2996966 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2025-04-08 20:22:24.543 INFO 2996966 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2025-04-08 20:22:24.544 INFO 2996966 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744114980, interval=60 2025-04-08 20:22:24.546 INFO 2996966 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2025-04-08 20:22:24.583 INFO 2996966 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2025-04-08 20:22:24.784 INFO 2996966 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2025-04-08 20:22:25.040 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 20:22:25.049 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 20:22:25.051 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:22:25.051 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:22:25.051 INFO 2996966 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2025-04-08 20:22:25.061 INFO 2996966 --- [main,,] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Create task, req=CreateTaskRequest(name=Clean expired log files on schedule, creatorId=200, groupId=2, scheduleType=SCHEDULE, scheduleRule=0 0 0 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, templateName=BackupScheduleCleanLogTemplateFactory, comments=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}, enable=true) 2025-04-08 20:22:25.123 INFO 2996966 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : Backup init, create system schedule task, task=TaskDefinition{id=1000000, name=Clean expired log files on schedule, creatorId=200, templateName=BackupScheduleCleanLogTemplateFactory, scheduleType=SCHEDULE, scheduleRule=0 0 0 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, nextRunTime=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}}. 2025-04-08 20:22:25.160 INFO 2996966 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2025-04-08 20:22:25.165 INFO 2996966 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2025-04-08 20:22:25.220 INFO 2996966 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2025-04-08 20:22:25.244 INFO 2996966 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2025-04-08 20:22:25.662 INFO 2996966 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2025-04-08 20:22:26.296 INFO 2996966 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2025-04-08 20:22:26.310 INFO 2996966 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2025-04-08 20:22:26.311 INFO 2996966 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2025-04-08 20:22:26.315 INFO 2996966 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2025-04-08 20:22:26.590 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:26.591 INFO 2996966 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:26.642 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$1989/1989275054@b932521 with [org.springframework.security.web.session.DisableEncodeUrlFilter@20b77ff2, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@6ed525c6, org.springframework.security.web.context.SecurityContextPersistenceFilter@76d209e0, org.springframework.security.web.header.HeaderWriterFilter@49511ae5, org.springframework.security.web.authentication.logout.LogoutFilter@10bdc0b, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@3eec66c2, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@2c44f613, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@37975486, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@702baca7, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@6df43735, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@46abe39f, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@779ce5cb, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@2dac1e16, org.springframework.security.web.session.SessionManagementFilter@1acba8d, org.springframework.security.web.access.ExceptionTranslationFilter@50ee0d0e, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@6d1fd768] 2025-04-08 20:22:26.678 INFO 2996966 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2025-04-08 20:22:26.678 INFO 2996966 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2025-04-08 20:22:26.680 INFO 2996966 --- [main,,] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Create task, req=CreateTaskRequest(name=Collect backup files size on schedule, creatorId=200, groupId=2, scheduleType=SCHEDULE, scheduleRule=0 0 */3 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, templateName=BackupScheduleCollectTemplateFactory, comments=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}, enable=true) 2025-04-08 20:22:26.690 INFO 2996966 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : Backup init, create system schedule task, task=TaskDefinition{id=1000001, name=Collect backup files size on schedule, creatorId=200, templateName=BackupScheduleCollectTemplateFactory, scheduleType=SCHEDULE, scheduleRule=0 0 */3 * * *, scheduleStartTime=null, scheduleEndTime=null, scheduleDurationUnit=Days, nextRunTime=null, arguments=Argument{parallelIdx=-1, stringMap={}, listMap={}}}. 2025-04-08 20:22:26.893 INFO 2996966 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2025-04-08 20:22:27.038 INFO 2996966 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2025-04-08 20:22:27.044 INFO 2996966 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2025-04-08 20:22:27.057 INFO 2996966 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2025-04-08 20:22:28.179 INFO 2996966 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2025-04-08 20:22:28.180 INFO 2996966 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2025-04-08 20:22:28.499 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@192271a1, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@51c30fac, org.springframework.security.web.context.SecurityContextPersistenceFilter@38cf85a4, org.springframework.security.web.header.HeaderWriterFilter@1c4a2252, org.springframework.security.web.csrf.CsrfFilter@6fc7aff5, org.springframework.security.web.authentication.logout.LogoutFilter@50c0c465, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@7e83aabc, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@36a17806, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@457561c8, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@6b347c96, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@2096868a, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@58909d73, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@523550b9, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@13ebf29e, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@13e8f545, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@5efd0dd8, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@1b429b3c, org.springframework.security.web.session.SessionManagementFilter@4ea09e29, org.springframework.security.web.access.ExceptionTranslationFilter@2c90f393, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@38137145] 2025-04-08 20:22:28.534 INFO 2996966 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2025-04-08 20:22:28.886 INFO 2996966 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2025-04-08 20:22:29.187 WARN 2996966 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2025-04-08 20:22:29.591 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:29.593 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:30.074 INFO 2996966 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2025-04-08 20:22:32.593 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:32.594 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:33.412 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/static/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/static/**'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff2']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff2'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.ttf']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.ttf'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.tif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.tif'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.gif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.gif'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.jpg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.jpg'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.png']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.png'] 2025-04-08 20:22:33.413 WARN 2996966 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:22:33.413 INFO 2996966 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2025-04-08 20:22:34.164 INFO 2996966 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2025-04-08 20:22:34.467 INFO 2996966 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2025-04-08 20:22:34.710 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=15 2025-04-08 20:22:34.710 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=inhibitAlarms, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=6 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkSuspendTenantCompactionStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.711 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=11 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterCgroupValue, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationServiceGroup, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.712 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationAutoReplaceStatusTask, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.729 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2025-04-08 20:22:34.729 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2025-04-08 20:22:34.729 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.735 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.735 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.735 INFO 2996966 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:22:34.742 INFO 2996966 --- [ocp-schedules-3,5f6367a20912bb45,377636fc8d6c09ce] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:22:34.742+08:00, clusters=[] 2025-04-08 20:22:34.742 INFO 2996966 --- [ocp-schedules-2,12fbbc4b27ae6596,697550a59fb40371] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:22:34.742+08:00, clusters=[] 2025-04-08 20:22:34.745 INFO 2996966 --- [ocp-schedules-22,7266dfb9a0cc5638,12a3df556ee08344] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:22:34.745 INFO 2996966 --- [ocp-schedules-10,768c350a1ca9bbeb,1b96bc979321e823] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:22:34.745 INFO 2996966 --- [ocp-schedules-4,c6b5eea5cf259607,41cb45581f27b59a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:22:34.745 INFO 2996966 --- [ocp-schedules-7,a9442b96b37158de,bfcd64a5f96b9099] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:22:34.750 INFO 2996966 --- [ocp-schedules-1,1d2ba3ea8a8988b9,514e1ef808659f39] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:22:34.753 WARN 2996966 --- [ocp-schedules-4,c6b5eea5cf259607,41cb45581f27b59a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncClusterInfo, confName=syncClusterInfo 2025-04-08 20:22:34.755 INFO 2996966 --- [ocp-schedules-11,6dee3dbc8f6ed983,4b2fd4d86a25aecf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:22:34.755 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2025-04-08 20:22:34.755 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2025-04-08 20:22:34.756 INFO 2996966 --- [ocp-schedules-5,1e1f3d9756d6ccfe,a2df3175154a4f60] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:22:34.756 INFO 2996966 --- [ocp-schedules-12,9d89abf97c107a53,fad9f7223a33306e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:22:34.756 INFO 2996966 --- [ocp-schedules-27,7f794c877de30a07,395598cbdc80ee76] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:22:34.757 INFO 2996966 --- [ocp-schedules-16,291d8df04b425841,55b011515df902f5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:22:34.760 INFO 2996966 --- [ocp-schedules-16,291d8df04b425841,55b011515df902f5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:22:34.761 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:22:34.761 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:22:34.761 ERROR 2996966 --- [ocp-schedules-11,6dee3dbc8f6ed983,4b2fd4d86a25aecf] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2025-04-08 20:22:34.761 INFO 2996966 --- [ocp-schedules-14,733e5d93dd0fb9bc,27c1ed60cf9241f4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:22:34.761 INFO 2996966 --- [ocp-schedules-9,4785f6c6b2561926,d5f7ce838dd9bcaa] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:22:34.762 INFO 2996966 --- [ocp-schedules-19,4fbf4ffc0bb28793,b5978915ddb13f29] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : binlog.cluster.parameter.blacklist= 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/root/ocp-server/log/ocp.log 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/oceanbaselog/sgkyocp/ocp-server.log 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2025-04-08 20:22:34.762 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2025-04-08 20:22:34.763 INFO 2996966 --- [ocp-schedules-20,b4852537c10654a8,795b267b1339cf33] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(2996966){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 2996966 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2025-04-08 20:22:34.763 INFO 2996966 --- [ocp-schedules-2,caf16caf2356232c,8352fc422b268373] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2025-04-08 20:22:34.763 INFO 2996966 --- [ocp-schedules-24,d6966279abce35e5,88891be1fb93bbd9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2025-04-08 20:22:34.763 INFO 2996966 --- [ocp-schedules-29,895a9273bf48e49a,026f439271cb1353] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.force.ssh.installation=false 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2025-04-08 20:22:34.763 INFO 2996966 --- [ocp-schedules-1,ecb3c8530803a1bc,0350c598fc7b4f18] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2025-04-08 20:22:34.763 INFO 2996966 --- [ocp-schedules-3,27bdfff228a9a718,41ca99300854e9ea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80},{"file_regex":"[a-z]+.log.[0-9]+.zst","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.enabled=true 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.run.internal=5m 2025-04-08 20:22:34.763 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.use.specific.http.ip=false 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full= 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2025-04-08 20:22:34.764 WARN 2996966 --- [ocp-schedules-3,27bdfff228a9a718,41ca99300854e9ea] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=checkClusterCgroupValue, confName=checkClusterCgroupValue 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2025-04-08 20:22:34.764 INFO 2996966 --- [ocp-schedules-11,6dee3dbc8f6ed983,4b2fd4d86a25aecf] com.alibaba.druid.pool.DruidDataSource : {dataSource-7} inited 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2025-04-08 20:22:34.764 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2025-04-08 20:22:34.766 INFO 2996966 --- [ocp-schedules-15,21caa409d3108bce,d28697969f335eda] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2025-04-08 20:22:34.770 INFO 2996966 --- [ocp-schedules-14,733e5d93dd0fb9bc,27c1ed60cf9241f4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:22:34.770 INFO 2996966 --- [ocp-schedules-9,4785f6c6b2561926,d5f7ce838dd9bcaa] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2025-04-08 20:22:34.770 INFO 2996966 --- [ocp-schedules-19,4fbf4ffc0bb28793,b5978915ddb13f29] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2025-04-08 20:22:34.770 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2025-04-08 20:22:34.771 WARN 2996966 --- [ocp-schedules-20,b4852537c10654a8,795b267b1339cf33] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncAllTenantInfo, confName=syncAllTenantInfo 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2025-04-08 20:22:34.771 INFO 2996966 --- [ocp-schedules-2,caf16caf2356232c,8352fc422b268373] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2025-04-08 20:22:34.771 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2025-04-08 20:22:34.777 INFO 2996966 --- [ocp-schedules-23,e307bf8cd7624c6d,99ea7bfd21fbb1fa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.object-storage-sdk.invoke.https=true 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2025-04-08 20:22:34.777 WARN 2996966 --- [ocp-async-8,0c72bbe4bec4bf4c,54b2f49a0b40006f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2025-04-08 20:22:34.777 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2025-04-08 20:22:34.778 ERROR 2996966 --- [ocp-schedules-3,27bdfff228a9a718,41ca99300854e9ea] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.checkClusterCgroupValue(ClusterSchedules.java:135) ... 15 common frames omitted 2025-04-08 20:22:34.778 ERROR 2996966 --- [ocp-schedules-4,c6b5eea5cf259607,41cb45581f27b59a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:80) ... 15 common frames omitted 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2025-04-08 20:22:34.778 ERROR 2996966 --- [ocp-schedules-20,b4852537c10654a8,795b267b1339cf33] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) ... 15 common frames omitted 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2025-04-08 20:22:34.778 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2025-04-08 20:22:34.779 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/root/ocp/ocp-server/lib 2025-04-08 20:22:34.781 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/root/software 2025-04-08 20:22:34.781 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2025-04-08 20:22:34.781 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2025-04-08 20:22:34.782 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2025-04-08 20:22:34.783 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2025-04-08 20:22:34.783 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2025-04-08 20:22:34.783 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2025-04-08 20:22:34.783 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2025-04-08 20:22:34.783 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2025-04-08 20:22:34.783 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2025-04-08 20:22:34.784 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2025-04-08 20:22:34.784 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2025-04-08 20:22:34.784 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2025-04-08 20:22:34.786 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/root/ocp/logs/ocp 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name= 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":3000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time= 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2025-04-08 20:22:34.787 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.rollup.type=MAX 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.service.type=LOCAL 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2025-04-08 20:22:34.788 INFO 2996966 --- [ocp-schedules-26,40a49f1e10b73992,6e060ea885dc1986] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2025-04-08 20:22:34.788 INFO 2996966 --- [ocp-schedules-26,40a49f1e10b73992,6e060ea885dc1986] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 20:22:34.788 INFO 2996966 --- [ocp-schedules-18,0c86f1e599d55ead,81db345285c28965] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=192.168.1.61 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2881 2025-04-08 20:22:34.788 INFO 2996966 --- [ocp-schedules-17,dece5a3278749f85,5b2a0a528d56bfdf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2025-04-08 20:22:34.788 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.batch.replace.arbitration.thread.core-pool-size=100 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist= 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-zA-Z_0-9-]{2,64}$ 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obshell.port=2886 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2025-04-08 20:22:34.789 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2025-04-08 20:22:34.790 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2025-04-08 20:22:34.791 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2025-04-08 20:22:34.792 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2025-04-08 20:22:34.793 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2025-04-08 20:22:34.794 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2025-04-08 20:22:34.794 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2025-04-08 20:22:34.794 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://192.168.1.61 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2025-04-08 20:22:34.795 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2025-04-08 20:22:34.796 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2025-04-08 20:22:34.797 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2025-04-08 20:22:34.797 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2025-04-08 20:22:34.797 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2025-04-08 20:22:34.799 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationAutoReplaceStatusTask.fixedDelay=60 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationServiceGroup.fixedDelay=10 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterCgroupValue.fixedDelay=60 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2025-04-08 20:22:34.803 INFO 2996966 --- [ocp-schedules-19,4fbf4ffc0bb28793,b5978915ddb13f29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:22:34.803 INFO 2996966 --- [ocp-schedules-19,4fbf4ffc0bb28793,b5978915ddb13f29] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:22:34.804 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:22:34.804 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:22:34.804 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:22:34.804 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:22:34.808 INFO 2996966 --- [ocp-schedules-2,caf16caf2356232c,8352fc422b268373] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:22:34.809 INFO 2996966 --- [ocp-schedules-6,f2626c6977763d40,fa45fa99ec291aca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:22:34.813 INFO 2996966 --- [ocp-async-5,a7de6844ea544f6b,a3c7ecca489a69f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:22:34.800 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkSuspendTenantCompactionStatus.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.oracle=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.5 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.5-20250319105844 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2025-04-08 20:22:34.814 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2025-04-08 20:22:34.815 INFO 2996966 --- [ocp-async-4,d4f475c21137a822,061093b6941281fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8080 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2025-04-08 20:22:34.815 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:22:34.816 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2025-04-08 20:22:34.817 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2025-04-08 20:22:34.818 INFO 2996966 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2025-04-08 20:22:34.821 INFO 2996966 --- [ocp-schedules-16,291d8df04b425841,55b011515df902f5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:22:34.821 INFO 2996966 --- [ocp-schedules-16,291d8df04b425841,55b011515df902f5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:22:34.822 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:22:34.822 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:22:34.822 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:22:34.822 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:22:34.827 INFO 2996966 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 268.296 seconds (JVM running for 269.103) 2025-04-08 20:22:34.829 INFO 2996966 --- [pool-overview-container-8,b9e371a848af9ce2,3a0af66f7d351c1d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:22:34.835 INFO 2996966 --- [ocp-schedules-30,4c860751e70d9e6c,0eec86b6a7f14e6b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:22:34.835 INFO 2996966 --- [pool-overview-container-5,e3d17027c11e15bd,5948ccf37e6d46c7] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:22:34.850 INFO 2996966 --- [ocp-schedules-25,feecf8ec49a15602,45baea7d50ab14cd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:22:34.850 INFO 2996966 --- [ocp-schedules-13,6f515ead54b0ff97,b6389f63e007ee24] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2025-04-08 20:22:34.857 INFO 2996966 --- [ocp-schedules-5,1e1f3d9756d6ccfe,a2df3175154a4f60] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:22:34.869 INFO 2996966 --- [ocp-schedules-21,2fc69ee27050a0db,9212b5d7573b7c91] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 20:22:34.874 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:22:34.877 INFO 2996966 --- [ocp-schedules-16,291d8df04b425841,55b011515df902f5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:22:34.877 INFO 2996966 --- [ocp-schedules-16,291d8df04b425841,55b011515df902f5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:22:34.877 INFO 2996966 --- [ocp-schedules-21,2fc69ee27050a0db,9212b5d7573b7c91] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 20:22:34.877 INFO 2996966 --- [ocp-schedules-21,2fc69ee27050a0db,9212b5d7573b7c91] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 20:22:34.877 INFO 2996966 --- [ocp-schedules-21,2fc69ee27050a0db,9212b5d7573b7c91] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 20:22:34.884 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:22:34.884 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:22:34.884 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:22:34.884 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:22:34.884 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:22:34.888 INFO 2996966 --- [ocp-schedules-13,6f515ead54b0ff97,b6389f63e007ee24] c.o.o.m.service.SeriesIdCleanService : HostIpSize=0, clusterSize=0, obproxyClusterSize=0 2025-04-08 20:22:34.889 INFO 2996966 --- [ocp-schedules-13,6f515ead54b0ff97,b6389f63e007ee24] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2025-04-08 20:22:34.889 INFO 2996966 --- [ocp-schedules-13,6f515ead54b0ff97,b6389f63e007ee24] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 20:22:34.889 INFO 2996966 --- [ocp-schedules-13,6f515ead54b0ff97,b6389f63e007ee24] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 20:22:34.889 INFO 2996966 --- [ocp-schedules-11,6dee3dbc8f6ed983,4b2fd4d86a25aecf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:22:34.890 INFO 2996966 --- [ocp-schedules-11,6dee3dbc8f6ed983,4b2fd4d86a25aecf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:22:34.896 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:22:34.897 INFO 2996966 --- [ocp-schedules-6,f2626c6977763d40,fa45fa99ec291aca] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:22:34.911 INFO 2996966 --- [ocp-schedules-25,feecf8ec49a15602,45baea7d50ab14cd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:22:34.916 INFO 2996966 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2025-04-08 20:22:34.919 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2025-04-08 20:22:34.922 INFO 2996966 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2025-04-08 20:22:34.939 INFO 2996966 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 493eecb387654b6f90f5379d8459859c. 2025-04-08 20:22:34.939 INFO 2996966 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2025-04-08 20:22:34.942 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT4M28.413560071S 2025-04-08 20:22:34.942 INFO 2996966 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2025-04-08 20:22:34.946 INFO 2996966 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2025-04-08 20:22:34.947 INFO 2996966 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2025-04-08 20:22:34.950 INFO 2996966 --- [ocp-async-8,efc6bdef6dd71ac3,30768764019b92ed] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [] 2025-04-08 20:22:34.950 INFO 2996966 --- [ocp-async-8,efc6bdef6dd71ac3,30768764019b92ed] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [] 2025-04-08 20:22:34.950 ERROR 2996966 --- [ocp-async-7,84f159d6ede606cd,eef4fb1c0e5b21fa] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2025-04-08 20:22:34.951 INFO 2996966 --- [ocp-async-7,84f159d6ede606cd,eef4fb1c0e5b21fa] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2025-04-08 20:22:34.951 INFO 2996966 --- [ocp-async-7,84f159d6ede606cd,eef4fb1c0e5b21fa] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2025-04-08 20:22:34.961 INFO 2996966 --- [ocp-async-7,84f159d6ede606cd,eef4fb1c0e5b21fa] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2025-04-08 20:22:34.967 WARN 2996966 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2025-04-08 20:22:34.967 INFO 2996966 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2025-04-08 20:22:34.967 INFO 2996966 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2025-04-08 20:22:35.216 INFO 2996966 --- [ocp-async-10,6f983215b8d72545,033a79bab609ecb1] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2025-04-08 20:22:35.240 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:22:35.348 INFO 2996966 --- [ocp-async-10,6f983215b8d72545,033a79bab609ecb1] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=623 2025-04-08 20:22:35.348 INFO 2996966 --- [ocp-async-10,6f983215b8d72545,033a79bab609ecb1] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2025-04-08 20:22:35.587 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.765052657S 2025-04-08 20:22:35.587 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:22:35.594 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:22:35.595 INFO 2996966 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:22:35.647 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:22:35.648 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:22:35.673 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:22:35.731 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:22:35.731 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:22:35.732 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:22:35.732 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:22:35.732 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:22:35.732 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:22:35.739 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:22:35.836 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] o.s.boot.SpringApplication : Started application in 1.032 seconds (JVM running for 270.112) 2025-04-08 20:22:35.836 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.032358001S 2025-04-08 20:22:36.170 INFO 2996966 --- [ocp-schedules-8,1cd1b86abc5be372,8d8ae95d39b7def3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:22:36.763 WARN 2996966 --- [ocp-async-1,f8c38dd9020a6da6,2a58088723bee33e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:22:36.764 INFO 2996966 --- [ocp-async-6,0c3f35d6c84da0a3,93ebb725870a5480] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:22:50.503 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:22:50.503 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:22:50.503 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:22:50.504 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:22:50.504 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:22:50.504 INFO 3009027 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2025-04-08 20:22:50.513 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:22:55.971 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:22:55.971 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:22:57.455 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1463 ms. Found 173 JPA repository interfaces. 2025-04-08 20:22:57.462 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:22:57.462 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:22:57.641 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 179 ms. Found 16 JPA repository interfaces. 2025-04-08 20:22:57.641 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:22:57.641 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:22:57.687 INFO 3009027 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 45 ms. Found 5 JPA repository interfaces. 2025-04-08 20:23:00.616 INFO 3009027 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=535a2406-c3b5-3b8b-8149-b06173f75dfb 2025-04-08 20:23:01.408 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7185e611' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.670 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7185e611' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.711 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c9e8a67' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.729 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7cca31fc' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.739 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7cca31fc' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.779 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration' of type [org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.803 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metadbConfiguration' of type [com.oceanbase.ocp.server.common.config.MetadbConfiguration$$EnhancerBySpringCGLIB$$d82e4643] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.850 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration' of type [org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.863 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.874 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'loadBalancerClientsDefaultsMappingsProvider' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration$$Lambda$660/1635975151] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.914 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultsBindHandlerAdvisor' of type [org.springframework.cloud.commons.config.DefaultsBindHandlerAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.934 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpClusterProperty.MetaDbProperties' of type [com.oceanbase.ocp.core.config.OcpClusterProperty$MetaDbProperties$$EnhancerBySpringCGLIB$$70729452] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:01.950 INFO 3009027 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:23:01.950 INFO 3009027 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2025-04-08 20:23:01.950 INFO 3009027 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:23:01.956 INFO 3009027 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2025-04-08 20:23:02.006 INFO 3009027 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2025-04-08 20:23:02.015 INFO 3009027 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:23:02.015 INFO 3009027 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:23:02.015 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaDataSource' of type [com.zaxxer.hikari.HikariDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.044 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.sql.init-org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties' of type [org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.070 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceScriptDatabaseInitializer' of type [org.springframework.boot.autoconfigure.sql.init.SqlDataSourceScriptDatabaseInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.098 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.111 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa.hibernate-org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.130 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.143 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hikariPoolDataSourceMetadataProvider' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration$$Lambda$428/1240412896] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.200 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.250 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaVendorAdapter' of type [org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.271 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'entityManagerFactoryBuilder' of type [org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:02.449 INFO 3009027 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 20:23:02.781 INFO 3009027 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2025-04-08 20:23:03.566 INFO 3009027 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2025-04-08 20:23:06.517 INFO 3009027 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:23:07.750 WARN 3009027 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2025-04-08 20:23:08.642 WARN 3009027 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 20:23:08.642 WARN 3009027 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 20:23:12.026 INFO 3009027 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 20:23:12.097 INFO 3009027 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:23:12.115 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.130 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [com.sun.proxy.$Proxy294] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.246 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#699e0bf0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.482 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'propertyManager.MonitorDbExtraProperties' of type [com.oceanbase.ocp.core.property.PropertyManager$MonitorDbExtraProperties$$EnhancerBySpringCGLIB$$2d2917e5] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.487 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'monitordbConfiguration' of type [com.oceanbase.ocp.server.common.config.MonitordbConfiguration$$EnhancerBySpringCGLIB$$15fd4ada] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.562 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpAzoneModelConverter' of type [com.oceanbase.ocp.ha.maz.OcpAzoneModelConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.582 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3823e936' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.583 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3823e936' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.584 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#39fb9d55' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.585 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#20520eb6' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.585 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#20520eb6' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.588 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1b0b843b' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.637 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.repository.config.JpaMetamodelMappingContextFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:12.644 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.mapping.JpaMetamodelMappingContext] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.587 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.607 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [com.sun.proxy.$Proxy300] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.678 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#673b5922' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.679 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#673b5922' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.680 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5b6626e5' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.680 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e96ff9a' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.681 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e96ff9a' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:13.683 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#518b5d87' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.046 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.047 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [com.sun.proxy.$Proxy304] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.068 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#614876bc' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.069 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#614876bc' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.069 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#249e48a3' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.070 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2547d1b8' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.070 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2547d1b8' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.071 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6256d908' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.096 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.097 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [com.sun.proxy.$Proxy305] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.117 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43b54374' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.117 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43b54374' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.118 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#73a8f9' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.118 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#78b9c5d8' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.119 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#78b9c5d8' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.120 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#37526ddf' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.153 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.177 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [com.sun.proxy.$Proxy306] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.228 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#515c6338' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.228 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#515c6338' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.230 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e3b72ad' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.230 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#451029fa' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.231 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#451029fa' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.233 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4af81941' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.262 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.263 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [com.sun.proxy.$Proxy307] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.332 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaTransactionManager' of type [org.springframework.orm.jpa.JpaTransactionManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.610 INFO 3009027 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action group type info: {"HOST_MANAGE":"HOST","ALL":"UNKNOWN","HOST_MAINTAIN":"HOST","ALL_READ":"UNKNOWN","OBPROXY_MAINTAIN":"OBPROXY","ORGANIZATION_READ":"ORGANIZATION","HOST_CREATE":"HOST","USER_MANAGE":"USER","ARBITRATION_READ":"ARBITRATION","ARBITRATION_CREATE":"ARBITRATION","BINLOG_MANAGE":"BINLOG","OBPROXY_MANAGE":"OBPROXY","HOST_READ":"HOST","BINLOG_READ":"BINLOG","AUDIT_READ":"AUDIT","USER_READ":"USER","ROLE_READ":"ROLE","CLUSTER_MANAGE":"CLUSTER","CLUSTER_CREATE":"CLUSTER","TENANT_MANAGE":"TENANT","CLUSTER_MAINTAIN":"CLUSTER","TENANT_MAINTAIN":"TENANT","CLUSTER_READ":"CLUSTER","BINLOG_MAINTAIN":"BINLOG","TENANT_READ":"TENANT","OBPROXY_READ":"OBPROXY","ARBITRATION_MAINTAIN":"ARBITRATION","ROLE_MANAGE":"ROLE","BINLOG_CREATE":"BINLOG","ARBITRATION_MANAGE":"ARBITRATION","OBPROXY_CREATE":"OBPROXY"} 2025-04-08 20:23:14.612 INFO 3009027 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action type info: {"ALARM:CREATE":"ALARM","SELFCURE:UPDATE":"SELFCURE","HOST:DELETE":"HOST","ROLE:CREATE":"ROLE","HOST:UPDATE":"HOST","CLUSTER:UPDATE":"CLUSTER","INSPECTION:UPDATE":"INSPECTION","CLUSTER:READ":"CLUSTER","PACKAGE:READ":"PACKAGE","ALARM:READ":"ALARM","TAG:DELETE":"TAG","BINLOG:CREATE":"BINLOG","ARBITRATION:CREATE":"ARBITRATION","SELFCURE:READ":"SELFCURE","*":"UNKNOWN","CLUSTER:DELETE":"CLUSTER","CREDENTIAL:DELETE":"CREDENTIAL","OBPROXY:READ":"OBPROXY","PROPERTY:READ":"PROPERTY","ARBITRATION:UPDATE":"ARBITRATION","OBPROXY:DELETE":"OBPROXY","TENANT:UPDATE":"TENANT","BINLOG:READ":"BINLOG","HOST:READ":"HOST","INSPECTION:CREATE":"INSPECTION","ALARM:UPDATE":"ALARM","PROPERTY:UPDATE":"PROPERTY","BINLOG:UPDATE":"BINLOG","PACKAGE:UPDATE":"PACKAGE","HOST:CREATE":"HOST","ROLE:DELETE":"ROLE","TAG:READ":"TAG","ROLE:READ":"ROLE","BINLOG:DELETE":"BINLOG","TASK:UPDATE":"TASK","USER:DELETE":"USER","CREDENTIAL:UPDATE":"CREDENTIAL","USER:UPDATE":"USER","TENANT:READ":"TENANT","TAG:UPDATE":"TAG","OBPROXY:CREATE":"OBPROXY","ARBITRATION:READ":"ARBITRATION","ROLE:UPDATE":"ROLE","ORGANIZATION:READ":"ORGANIZATION","TENANT:DELETE":"TENANT","USER:READ":"USER","AUDIT:READ":"AUDIT","TASK:READ":"TASK","TAG:CREATE":"TAG","INSPECTION:READ":"INSPECTION","ARBITRATION:DELETE":"ARBITRATION","USER:CREATE":"USER","PACKAGE:DELETE":"PACKAGE","ALARM:DELETE":"ALARM","CREDENTIAL:READ":"CREDENTIAL","OBPROXY:UPDATE":"OBPROXY","CREDENTIAL:CREATE":"CREDENTIAL","CLUSTER:CREATE":"CLUSTER"} 2025-04-08 20:23:14.612 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionMetaHelper' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ActionMetaHelper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.637 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultPerfConfiguration' of type [com.oceanbase.ocp.server.common.config.DefaultPerfConfiguration$$EnhancerBySpringCGLIB$$b55198e8] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.672 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'authenticationFacadeImpl' of type [com.oceanbase.ocp.core.security.AuthenticationFacadeImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.697 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#32f40c92' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.697 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#32f40c92' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.698 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#77810475' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.698 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#239e7554' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.699 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#239e7554' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.700 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6bbced25' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.784 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.785 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [com.sun.proxy.$Proxy310] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.801 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOperator' of type [com.oceanbase.ocp.core.security.organization.operator.HostOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.820 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1f15fb2c' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.820 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1f15fb2c' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.821 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#527c4caa' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.821 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#16d76df7' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.822 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#16d76df7' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.823 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6cbe5b2a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.885 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.886 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [com.sun.proxy.$Proxy312] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.907 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObClusterOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.926 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#10218fdf' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.927 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#10218fdf' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.928 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5258833e' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.928 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#58bb8b59' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.928 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#58bb8b59' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:14.929 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7552c30e' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.003 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.004 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [com.sun.proxy.$Proxy313] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.038 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObproxyOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.057 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#eae8dd' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.057 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#eae8dd' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.058 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7859c88f' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.058 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#57c7e57' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.058 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#57c7e57' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.060 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74fd6fed' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.097 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.098 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [com.sun.proxy.$Proxy315] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.110 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObTenantOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.125 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ee7e602' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.126 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ee7e602' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.127 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7139ff5a' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.127 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6dcdacf2' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.127 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6dcdacf2' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.129 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#553f850d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.188 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.189 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [com.sun.proxy.$Proxy316] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.210 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOperator' of type [com.oceanbase.ocp.core.security.organization.operator.UserOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.227 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#597ca350' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.227 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#597ca350' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.228 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3e2544e9' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.228 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#87f336d' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.229 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#87f336d' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.230 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27241127' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.255 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.256 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [com.sun.proxy.$Proxy318] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.275 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ArbitrationOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.290 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#15f4dffe' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.291 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#15f4dffe' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.291 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27d8008a' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.292 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c29f8aa' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.292 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c29f8aa' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.293 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#e158c25' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.329 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.330 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [com.sun.proxy.$Proxy319] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.343 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogOperator' of type [com.oceanbase.ocp.core.security.organization.operator.BinlogOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.345 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceManager' of type [com.oceanbase.ocp.core.security.organization.ResourceManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.360 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#45a2d27c' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.360 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#45a2d27c' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.361 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75a9e1ad' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.361 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d3c799e' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.361 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d3c799e' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.363 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4008caaf' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.398 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.399 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [com.sun.proxy.$Proxy320] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.427 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#64a07d55' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.428 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#64a07d55' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.429 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#67aadf64' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.429 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b31c0ba' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.429 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b31c0ba' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.430 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2eac48ea' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.449 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.450 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [com.sun.proxy.$Proxy322] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.478 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#73b49ca4' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.479 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#73b49ca4' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.480 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#362f5054' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.481 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#65d8f9d2' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.481 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#65d8f9d2' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.484 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#68be702f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.515 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.515 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [com.sun.proxy.$Proxy323] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.536 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'getOrganizationService' of type [com.oceanbase.ocp.core.security.organization.OrganizationServiceImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.565 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4611a950' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.565 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4611a950' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.566 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#52333498' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.566 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#524206cb' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.566 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#524206cb' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.567 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#9caecf7' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.673 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.674 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [com.sun.proxy.$Proxy324] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.717 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#59708aaa' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.717 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#59708aaa' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.718 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7b7d5d8d' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.718 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#161267b6' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.718 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#161267b6' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.719 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7b742ec1' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.747 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.748 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [com.sun.proxy.$Proxy325] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.772 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2bedd126' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.772 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2bedd126' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.773 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6291cdb4' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.773 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#50406145' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.774 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#50406145' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.775 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#34c3a04a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.831 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.832 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [com.sun.proxy.$Proxy326] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.864 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6a43c847' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.864 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6a43c847' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.865 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4b3816c' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.865 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b7cd454' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.866 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b7cd454' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.867 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4297f992' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.894 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.895 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [com.sun.proxy.$Proxy327] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:15.920 INFO 3009027 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:23:15.996 INFO 3009027 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:23:15.996 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.017 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#459dea3e' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.017 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#459dea3e' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.018 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#44ebf052' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.018 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#179683c3' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.018 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#179683c3' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.019 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3a2486d9' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.059 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.060 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [com.sun.proxy.$Proxy330] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.087 INFO 3009027 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:23:16.099 INFO 3009027 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:23:16.099 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obProxyQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObProxyQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.119 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#45cdfb42' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.119 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#45cdfb42' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.120 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#12e0fe90' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.120 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#168d5f34' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.121 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#168d5f34' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.122 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#c4b2c69' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.152 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.152 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [com.sun.proxy.$Proxy331] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.179 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4ae707c' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.179 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4ae707c' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.180 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2f8892c5' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.180 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c7108ce' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.180 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c7108ce' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.181 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#77b3a4c0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.216 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.216 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [com.sun.proxy.$Proxy332] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.236 INFO 3009027 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:23:16.256 INFO 3009027 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:23:16.256 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogQueryCache' of type [com.oceanbase.ocp.core.cache.helper.BinlogQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.259 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceCache' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ResourceCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.261 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.272 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'adminResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.281 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'clusterResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.291 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.300 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.309 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.319 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'tenantResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.322 INFO 3009027 --- [main,,] c.o.o.c.s.a.r.ResourceSupplierManager : Resource supplier list: [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl@53477d08, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl@4d583069, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl@29863c20, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl@4cc93e77, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl@4a8afcae, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl@26bc84dc, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl@690d8a06] 2025-04-08 20:23:16.322 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceSupplierManager' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.ResourceSupplierManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.349 INFO 3009027 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$e43e1570] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:23:16.365 INFO 3009027 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2025-04-08 20:23:16.856 INFO 3009027 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=192.168.1.61:8080, name=b88d8e1c7d 2025-04-08 20:23:16.896 WARN 3009027 --- [main,,] c.o.o.c.d.server.DefaultServerUpdater : Deleted expired server: DistributedServer{id=192.168.1.61:8080, name=b88d8e1c7d, expireInSeconds=30, createTime=2025-04-08T20:21:05+08:00, updateTime=2025-04-08T20:22:30+08:00, expireTime=2025-04-08T20:23+08:00, isActive=false} 2025-04-08 20:23:16.916 INFO 3009027 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 192.168.1.61:8080 2025-04-08 20:23:17.799 INFO 3009027 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2025-04-08 20:23:17.832 INFO 3009027 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2025-04-08 20:23:17.832 INFO 3009027 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.98] 2025-04-08 20:23:17.996 INFO 3009027 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2025-04-08 20:23:17.996 INFO 3009027 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 27483 ms 2025-04-08 20:23:18.382 INFO 3009027 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2025-04-08 20:23:18.629 INFO 3009027 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:23:18.629 INFO 3009027 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2025-04-08 20:23:18.629 INFO 3009027 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:23:18.659 INFO 3009027 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:23:18.659 INFO 3009027 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:23:19.166 INFO 3009027 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2025-04-08 20:23:19.176 INFO 3009027 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : host 192.168.1.61, port 2881, database monitor_database 2025-04-08 20:23:19.178 INFO 3009027 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:23:19.178 INFO 3009027 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2025-04-08 20:23:19.178 INFO 3009027 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:23:19.178 INFO 3009027 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2025-04-08 20:23:19.181 INFO 3009027 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2025-04-08 20:23:19.182 INFO 3009027 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:23:19.182 INFO 3009027 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:23:19.800 INFO 3009027 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 20:23:22.031 INFO 3009027 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:23:22.420 INFO 3009027 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 20:23:22.421 INFO 3009027 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:23:35.892 INFO 3009027 --- [main,,] org.reflections.Reflections : Reflections took 4752 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 20:23:40.336 INFO 3009027 --- [main,,] org.reflections.Reflections : Reflections took 4384 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 20:23:44.831 INFO 3009027 --- [main,,] org.reflections.Reflections : Reflections took 4483 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 20:23:44.845 INFO 3009027 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2025-04-08 20:23:45.015 INFO 3009027 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2025-04-08 20:23:45.502 INFO 3009027 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2025-04-08 20:23:46.070 INFO 3009027 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2025-04-08 20:23:46.094 INFO 3009027 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2025-04-08 20:23:46.094 INFO 3009027 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2025-04-08 20:23:46.972 INFO 3009027 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=arbitration_group_auto_replace_failed,ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,arbitration_auto_replace_operation_info,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_merge_error,ob_tenant_name_mismatch, 2025-04-08 20:23:48.005 INFO 3009027 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table switch time is 1744114820 2025-04-08 20:23:49.246 INFO 3009027 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:23:49.273 INFO 3009027 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:23:49.284 INFO 3009027 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2025-04-08 20:23:50.344 INFO 3009027 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion 2025-04-08 20:23:50.350 INFO 3009027 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:23:50.350 INFO 3009027 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion 2025-04-08 20:23:50.350 INFO 3009027 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:23:50.351 INFO 3009027 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 20:23:50.846 INFO 3009027 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/root/ocp-server/alarm 2025-04-08 20:23:51.022 INFO 3009027 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2025-04-08 20:23:51.069 INFO 3009027 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=192.168.1.61:3009027 2025-04-08 20:23:51.144 INFO 3009027 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 20:23:51.262 INFO 3009027 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 20:23:51.283 INFO 3009027 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2025-04-08 20:23:51.326 INFO 3009027 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2025-04-08 20:23:51.433 INFO 3009027 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=1, heartbeatTime=2025-04-08T20:22:35+08:00, now=2025-04-08T20:23:51.432+08:00 2025-04-08 20:23:51.433 INFO 3009027 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=2, heartbeatTime=2025-04-08T20:22:35+08:00, now=2025-04-08T20:23:51.432+08:00 2025-04-08 20:23:51.531 INFO 3009027 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=192.168.1.61:8080, consumerId=2, resources=[ShardingSlice(consumerId=2, type=DEFAULT, resourceIds=[18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 80, 81, 106, 107, 250, 251, 252, 253, 254])] 2025-04-08 20:23:51.621 INFO 3009027 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2025-04-08 20:23:51.624 INFO 3009027 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2025-04-08 20:23:51.988 INFO 3009027 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2025-04-08 20:23:52.530 INFO 3009027 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2025-04-08 20:23:52.581 INFO 3009027 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2025-04-08 20:23:52.797 INFO 3009027 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/root/software 2025-04-08 20:23:55.360 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2025-04-08 20:23:55.360 INFO 3009027 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2025-04-08 20:23:55.362 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2025-04-08 20:23:55.376 INFO 3009027 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2025-04-08 20:23:55.379 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2025-04-08 20:23:55.379 INFO 3009027 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2025-04-08 20:23:55.380 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2025-04-08 20:23:55.380 INFO 3009027 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2025-04-08 20:23:55.381 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2025-04-08 20:23:55.382 INFO 3009027 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2025-04-08 20:23:55.383 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2025-04-08 20:23:55.383 INFO 3009027 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2025-04-08 20:23:55.384 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2025-04-08 20:23:55.384 INFO 3009027 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2025-04-08 20:23:55.385 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2025-04-08 20:23:55.388 INFO 3009027 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2025-04-08 20:23:55.389 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2025-04-08 20:23:55.389 INFO 3009027 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2025-04-08 20:23:55.390 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2025-04-08 20:23:55.390 INFO 3009027 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2025-04-08 20:23:55.391 INFO 3009027 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2025-04-08 20:23:55.391 INFO 3009027 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2025-04-08 20:23:56.653 INFO 3009027 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=1, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@18a60105 2025-04-08 20:23:56.653 INFO 3009027 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@6b5986b8 2025-04-08 20:23:56.655 INFO 3009027 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:23:56.655 INFO 3009027 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:23:57.543 INFO 3009027 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:23:57.544 INFO 3009027 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:23:57.682 INFO 3009027 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:23:57.804 INFO 3009027 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:24:00.576 INFO 3009027 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:24:00.576 INFO 3009027 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 20:24:00.591 INFO 3009027 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:24:00.591 INFO 3009027 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 20:24:01.932 INFO 3009027 --- [main,,] org.reflections.Reflections : Reflections took 4431 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 20:24:01.943 INFO 3009027 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2025-04-08 20:24:01.944 INFO 3009027 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2025-04-08 20:24:01.944 INFO 3009027 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2025-04-08 20:24:01.944 INFO 3009027 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2025-04-08 20:24:01.944 INFO 3009027 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2025-04-08 20:24:01.962 INFO 3009027 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on b88d8e1c7d 2025-04-08 20:24:02.989 INFO 3009027 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2025-04-08 20:24:03.664 INFO 3009027 --- [main,,] c.o.o.s.g.prefs.PrefsParameterService : Start loading prefs template. 2025-04-08 20:24:04.132 INFO 3009027 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Begin init binlog cluster parameter info. 2025-04-08 20:24:04.181 INFO 3009027 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Finish init binlog cluster parameter info, parameter blacklist: [] 2025-04-08 20:24:04.233 INFO 3009027 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Begin init binlog instance parameter info. 2025-04-08 20:24:04.240 INFO 3009027 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Finish init binlog instance parameter info, parameter blacklist: [] 2025-04-08 20:24:05.173 INFO 3009027 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2025-04-08 20:24:06.283 INFO 3009027 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2025-04-08 20:24:06.316 INFO 3009027 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB, valid=true 2025-04-08 20:24:06.316 INFO 3009027 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Sensitive data encryption key is legal. 2025-04-08 20:24:06.430 INFO 3009027 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2025-04-08 20:24:06.707 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 20:24:06.740 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 20:24:06.763 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:24:06.771 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:24:08.063 INFO 3009027 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 192.168.1.61:8080 2025-04-08 20:24:08.073 INFO 3009027 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2025-04-08 20:24:08.118 WARN 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 20:24:08.118 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 20:24:08.510 INFO 3009027 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2025-04-08 20:24:08.523 INFO 3009027 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2025-04-08 20:24:10.168 INFO 3009027 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/root/ocp/ocp-server/lib 2025-04-08 20:24:10.168 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2025-04-08 20:24:10.197 INFO 3009027 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=obproxyd.sh, objectId=1 2025-04-08 20:24:10.218 INFO 3009027 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2025-04-08 20:24:10.244 INFO 3009027 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh.sha1sum 2025-04-08 20:24:10.244 INFO 3009027 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh 2025-04-08 20:24:10.244 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2025-04-08 20:24:10.281 INFO 3009027 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=7 2025-04-08 20:24:10.322 INFO 3009027 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=7, name=obproxyd.sh 2025-04-08 20:24:10.335 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 20:24:10.358 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 20:24:10.358 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 20:24:10.361 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 20:24:10.361 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:24:10.364 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:24:10.364 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:24:10.367 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:24:10.368 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 20:24:10.370 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 20:24:10.372 INFO 3009027 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.5-20250319105844 2025-04-08 20:24:10.378 INFO 3009027 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.5-20250319105844, updatedValue=4.3.5-20250319105844, runningValue=4.3.5-20250319105844 2025-04-08 20:24:10.421 INFO 3009027 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"update_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition","timeoutNs":0},"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event","timeoutNs":0},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception","timeoutNs":0},"cleanStorageObjectBlock":{"batchIntervalMillis":10000,"batchSize":10,"condition":"file_id not in (select id from storage_object_meta)","priority":0,"tableName":"storage_object_block","timeoutNs":0}} 2025-04-08 20:24:10.870 INFO 3009027 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2025-04-08 20:24:11.114 INFO 3009027 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2025-04-08 20:24:11.116 INFO 3009027 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2025-04-08 20:24:11.119 WARN 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 20:24:11.119 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 20:24:12.452 INFO 3009027 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2025-04-08 20:24:12.567 INFO 3009027 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2025-04-08 20:24:12.569 INFO 3009027 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:12.570 INFO 3009027 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2025-04-08 20:24:12.605 INFO 3009027 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2025-04-08 20:24:12.802 INFO 3009027 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2025-04-08 20:24:13.071 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=CMDB_SYNCER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.074 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=MC_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.076 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PERF_SQL_ROLLUP, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.077 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.078 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=RATE_LIMIT, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.080 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.081 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.083 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=TASK_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:22:01+08:00, updateTime=2025-04-08T20:22:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:24:13.083 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.099 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.104 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.106 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.108 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.110 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.111 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.113 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:24:13.115 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 20:24:13.123 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 20:24:13.125 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:24:13.126 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:24:13.126 INFO 3009027 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2025-04-08 20:24:13.225 INFO 3009027 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2025-04-08 20:24:13.230 INFO 3009027 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2025-04-08 20:24:13.279 INFO 3009027 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2025-04-08 20:24:13.328 INFO 3009027 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2025-04-08 20:24:13.540 INFO 3009027 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=0, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@5bd02d4b 2025-04-08 20:24:13.541 INFO 3009027 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:24:13.735 INFO 3009027 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2025-04-08 20:24:14.120 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 gets the leadership... 2025-04-08 20:24:14.408 INFO 3009027 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2025-04-08 20:24:14.422 INFO 3009027 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2025-04-08 20:24:14.424 INFO 3009027 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2025-04-08 20:24:14.428 INFO 3009027 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2025-04-08 20:24:14.793 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$1981/1919229837@e4e9bc2 with [org.springframework.security.web.session.DisableEncodeUrlFilter@753348c3, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@726fcd5b, org.springframework.security.web.context.SecurityContextPersistenceFilter@28882179, org.springframework.security.web.header.HeaderWriterFilter@3819e94a, org.springframework.security.web.authentication.logout.LogoutFilter@40cf22da, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@57431060, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@2da901c1, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@423321e7, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@4e3a29ef, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@65d1e0fa, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@5ed0c4ae, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@69acf3f2, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@1b5e058, org.springframework.security.web.session.SessionManagementFilter@c071ede, org.springframework.security.web.access.ExceptionTranslationFilter@5493f4d, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@1df48c2] 2025-04-08 20:24:14.804 INFO 3009027 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2025-04-08 20:24:14.804 INFO 3009027 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2025-04-08 20:24:15.024 INFO 3009027 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2025-04-08 20:24:15.161 INFO 3009027 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2025-04-08 20:24:15.167 INFO 3009027 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2025-04-08 20:24:15.180 INFO 3009027 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2025-04-08 20:24:16.314 INFO 3009027 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2025-04-08 20:24:16.315 INFO 3009027 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2025-04-08 20:24:16.547 INFO 3009027 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:24:16.576 INFO 3009027 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:24:16.624 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@262891c7, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@308b51c1, org.springframework.security.web.context.SecurityContextPersistenceFilter@3d8a6cd1, org.springframework.security.web.header.HeaderWriterFilter@20aa43f0, org.springframework.security.web.csrf.CsrfFilter@504d1341, org.springframework.security.web.authentication.logout.LogoutFilter@318665cc, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@6538889c, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@1bcaf722, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@663e8537, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@4b993cf7, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@6f46cf06, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@32183813, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@50dc313e, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@103e1cf0, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@19eff5ae, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@6ba880a0, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@3ab367bd, org.springframework.security.web.session.SessionManagementFilter@18459bbf, org.springframework.security.web.access.ExceptionTranslationFilter@4efd4781, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@24af4f9c] 2025-04-08 20:24:16.664 INFO 3009027 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2025-04-08 20:24:17.086 INFO 3009027 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2025-04-08 20:24:17.120 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:17.129 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:17.403 WARN 3009027 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2025-04-08 20:24:18.308 INFO 3009027 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2025-04-08 20:24:19.554 INFO 3009027 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:24:19.554 INFO 3009027 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2025-04-08 20:24:20.129 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:20.130 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.718 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2025-04-08 20:24:21.718 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/static/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/static/**'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff2']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff2'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.ttf']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.ttf'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.tif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.tif'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.gif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.gif'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.jpg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.jpg'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.png']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.png'] 2025-04-08 20:24:21.719 WARN 3009027 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:24:21.719 INFO 3009027 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2025-04-08 20:24:22.455 INFO 3009027 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2025-04-08 20:24:22.772 INFO 3009027 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2025-04-08 20:24:23.015 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=15 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=inhibitAlarms, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.016 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=6 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkSuspendTenantCompactionStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=11 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterCgroupValue, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationServiceGroup, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.017 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationAutoReplaceStatusTask, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.037 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2025-04-08 20:24:23.037 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2025-04-08 20:24:23.037 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.044 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.044 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.044 INFO 3009027 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:24:23.050 INFO 3009027 --- [ocp-schedules-3,6977e15450bd1733,ce5f4ffe050f1749] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:24:23.049+08:00, clusters=[] 2025-04-08 20:24:23.050 INFO 3009027 --- [ocp-schedules-2,4205567ecdb7fc53,a1b0da14adcea943] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:24:23.049+08:00, clusters=[] 2025-04-08 20:24:23.056 INFO 3009027 --- [ocp-schedules-1,9f047a9e90f05410,ff39f57eda6fab91] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:24:23.058 INFO 3009027 --- [ocp-schedules-7,39c79109c2372d40,49bbab363fc1b9ca] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:24:23.059 INFO 3009027 --- [ocp-schedules-27,273fe537567ef64a,8ee2fde306b95cfd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:24:23.060 INFO 3009027 --- [ocp-schedules-20,3eddb372666dfdbb,599bed1ebf024f7b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:24:23.063 INFO 3009027 --- [ocp-schedules-5,f2c8414720ab36c5,b6e34ecd5830103f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:24:23.064 WARN 3009027 --- [ocp-schedules-5,f2c8414720ab36c5,b6e34ecd5830103f] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncClusterInfo, confName=syncClusterInfo 2025-04-08 20:24:23.066 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2025-04-08 20:24:23.066 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2025-04-08 20:24:23.067 INFO 3009027 --- [ocp-schedules-10,95d04568de2d6982,9b221d3557139d38] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:24:23.069 INFO 3009027 --- [ocp-schedules-11,d85cb2b1491df25d,3551fe6814b1520d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:24:23.069 ERROR 3009027 --- [ocp-schedules-10,95d04568de2d6982,9b221d3557139d38] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2025-04-08 20:24:23.070 INFO 3009027 --- [ocp-schedules-28,1b7ad73ad48b75ea,ffa031499a30b7e1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:24:23.070 INFO 3009027 --- [ocp-schedules-10,95d04568de2d6982,9b221d3557139d38] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} inited 2025-04-08 20:24:23.071 INFO 3009027 --- [ocp-schedules-14,16b768c869ca1190,d5d68fe023a11d44] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:24:23.071 INFO 3009027 --- [ocp-schedules-14,16b768c869ca1190,d5d68fe023a11d44] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:24:23.073 ERROR 3009027 --- [ocp-schedules-5,f2c8414720ab36c5,b6e34ecd5830103f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:80) ... 15 common frames omitted 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-18,daa243ea3575fb60,bb234c3c9d590e7d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-5,b056f24aa4a3bee8,5aecc369187e7584] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-17,af474716214b397e,35729d862e174aaf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-12,4e30399b38500091,1e0c2ea2fd381a04] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-15,41844fd4e9580aca,4d6a5a48a6041465] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-4,64d22f94ebf6f26a,956928578f8e7f3f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 20:24:23.073 INFO 3009027 --- [ocp-schedules-30,ec292e04f4d2cd18,5fb215ab52ae595f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:24:23.074 INFO 3009027 --- [ocp-schedules-21,aa62b429d41d0c13,271b1c4069c69e7e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:24:23.074 INFO 3009027 --- [ocp-schedules-21,aa62b429d41d0c13,271b1c4069c69e7e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : binlog.cluster.parameter.blacklist= 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2025-04-08 20:24:23.074 INFO 3009027 --- [ocp-schedules-6,b9da5f67a9b912c0,dfd75837bf75766e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/root/ocp-server/log/ocp.log 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/oceanbaselog/sgkyocp/ocp-server.log 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2025-04-08 20:24:23.074 INFO 3009027 --- [ocp-schedules-19,863226ea86afc2d1,cb1bbbee8afc52d9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(3009027){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2025-04-08 20:24:23.074 WARN 3009027 --- [ocp-schedules-6,b9da5f67a9b912c0,dfd75837bf75766e] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=checkClusterCgroupValue, confName=checkClusterCgroupValue 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 3009027 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2025-04-08 20:24:23.074 ERROR 3009027 --- [ocp-schedules-6,b9da5f67a9b912c0,dfd75837bf75766e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.checkClusterCgroupValue(ClusterSchedules.java:135) ... 15 common frames omitted 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2025-04-08 20:24:23.074 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2025-04-08 20:24:23.075 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2025-04-08 20:24:23.075 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2025-04-08 20:24:23.075 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2025-04-08 20:24:23.075 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2025-04-08 20:24:23.075 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.force.ssh.installation=false 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80},{"file_regex":"[a-z]+.log.[0-9]+.zst","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.enabled=true 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.run.internal=5m 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.use.specific.http.ip=false 2025-04-08 20:24:23.076 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full=4.3.5-20250319105844 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2025-04-08 20:24:23.077 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.application.uuid=493eecb387654b6f90f5379d8459859c 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.object-storage-sdk.invoke.https=true 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2025-04-08 20:24:23.078 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2025-04-08 20:24:23.079 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2025-04-08 20:24:23.079 INFO 3009027 --- [ocp-schedules-5,b056f24aa4a3bee8,5aecc369187e7584] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:24:23.079 INFO 3009027 --- [ocp-schedules-18,daa243ea3575fb60,bb234c3c9d590e7d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:24:23.079 WARN 3009027 --- [ocp-schedules-17,af474716214b397e,35729d862e174aaf] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncAllTenantInfo, confName=syncAllTenantInfo 2025-04-08 20:24:23.079 ERROR 3009027 --- [ocp-schedules-17,af474716214b397e,35729d862e174aaf] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) ... 15 common frames omitted 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/root/ocp/ocp-server/lib 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/root/software 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2025-04-08 20:24:23.080 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2025-04-08 20:24:23.081 INFO 3009027 --- [ocp-schedules-16,441ec77ce3ecf300,f0649447fcdbc1f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2025-04-08 20:24:23.081 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2025-04-08 20:24:23.081 INFO 3009027 --- [ocp-schedules-12,4e30399b38500091,1e0c2ea2fd381a04] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/root/ocp/logs/ocp 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name= 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":3000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2025-04-08 20:24:23.082 INFO 3009027 --- [ocp-async-4,ff3be13874d77412,1aaf11b72ee824c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time=1744114820 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.rollup.type=MAX 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.service.type=LOCAL 2025-04-08 20:24:23.082 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2025-04-08 20:24:23.082 INFO 3009027 --- [ocp-async-5,319d004a9578db49,62c8b3399fa92f21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:23.083 INFO 3009027 --- [ocp-schedules-15,41844fd4e9580aca,4d6a5a48a6041465] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:24:23.084 INFO 3009027 --- [ocp-schedules-9,64e938d4c876b4c2,b6b113e6f88a6986] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 20:24:23.085 INFO 3009027 --- [ocp-schedules-1,5a8af57f1a2998ce,943cf03ab51e1c0b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:24:23.085 INFO 3009027 --- [ocp-schedules-25,6364fbff79c84dfc,75c13c1b60f8a901] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:24:23.085 INFO 3009027 --- [ocp-schedules-26,8704d73c84b4bbdf,d7e5b9a4c8578d99] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2025-04-08 20:24:23.088 INFO 3009027 --- [ocp-schedules-9,64e938d4c876b4c2,b6b113e6f88a6986] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2025-04-08 20:24:23.088 INFO 3009027 --- [ocp-schedules-9,64e938d4c876b4c2,b6b113e6f88a6986] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2025-04-08 20:24:23.088 INFO 3009027 --- [ocp-schedules-9,64e938d4c876b4c2,b6b113e6f88a6986] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=192.168.1.61 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2025-04-08 20:24:23.088 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2881 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2025-04-08 20:24:23.089 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2025-04-08 20:24:23.090 WARN 3009027 --- [ocp-async-7,d9dd1c912c2b8954,2f9531ea8ec1be70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.batch.replace.arbitration.thread.core-pool-size=100 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist= 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-zA-Z_0-9-]{2,64}$ 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2025-04-08 20:24:23.090 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obshell.port=2886 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2025-04-08 20:24:23.091 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2025-04-08 20:24:23.092 INFO 3009027 --- [ocp-schedules-3,dcd11f1a3406db6b,4080748dd394d770] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2025-04-08 20:24:23.092 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2025-04-08 20:24:23.093 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2025-04-08 20:24:23.094 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2025-04-08 20:24:23.095 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2025-04-08 20:24:23.099 INFO 3009027 --- [ocp-schedules-8,ba0696b9d9eaa1ba,05c54a97105cb167] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:24:23.099 INFO 3009027 --- [ocp-schedules-11,5236f879cb44a561,fc063aafaa0d37ab] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:24:23.101 INFO 3009027 --- [ocp-schedules-22,cec4f0281ab4f645,86e607d6e8bc77fa] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:24:23.110 INFO 3009027 --- [ocp-schedules-13,fea52acd7cdedf09,601b11400848330b] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2025-04-08 20:24:23.110 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:24:23.110 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:24:23.110 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:24:23.110 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2025-04-08 20:24:23.111 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://192.168.1.61 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2025-04-08 20:24:23.112 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.fixedDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [ocp-schedules-18,daa243ea3575fb60,bb234c3c9d590e7d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:24:23.113 INFO 3009027 --- [ocp-schedules-18,daa243ea3575fb60,bb234c3c9d590e7d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2025-04-08 20:24:23.113 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2025-04-08 20:24:23.114 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationAutoReplaceStatusTask.fixedDelay=60 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationServiceGroup.fixedDelay=10 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterCgroupValue.fixedDelay=60 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2025-04-08 20:24:23.115 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2025-04-08 20:24:23.116 INFO 3009027 --- [ocp-schedules-12,4e30399b38500091,1e0c2ea2fd381a04] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkSuspendTenantCompactionStatus.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2025-04-08 20:24:23.116 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.oracle=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.5 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.5-20250319105844 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2025-04-08 20:24:23.117 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8080 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2025-04-08 20:24:23.118 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:24:23.118 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2025-04-08 20:24:23.118 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2025-04-08 20:24:23.118 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2025-04-08 20:24:23.118 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2025-04-08 20:24:23.119 INFO 3009027 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2025-04-08 20:24:23.120 INFO 3009027 --- [ocp-schedules-3,dcd11f1a3406db6b,4080748dd394d770] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:24:23.126 INFO 3009027 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 103.785 seconds (JVM running for 104.582) 2025-04-08 20:24:23.130 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:23.133 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:23.135 INFO 3009027 --- [ocp-schedules-10,95d04568de2d6982,9b221d3557139d38] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:24:23.135 INFO 3009027 --- [ocp-schedules-10,95d04568de2d6982,9b221d3557139d38] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:24:23.138 INFO 3009027 --- [ocp-schedules-13,fea52acd7cdedf09,601b11400848330b] c.o.o.m.service.SeriesIdCleanService : HostIpSize=0, clusterSize=0, obproxyClusterSize=0 2025-04-08 20:24:23.157 INFO 3009027 --- [pool-overview-container-6,4f334ba89806d935,180504a2dfc2a61d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:24:23.157 INFO 3009027 --- [pool-overview-container-5,d9df681426c57826,5b8a88dcbbd0df65] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:24:23.158 INFO 3009027 --- [ocp-schedules-14,16b768c869ca1190,d5d68fe023a11d44] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:24:23.158 INFO 3009027 --- [ocp-schedules-14,16b768c869ca1190,d5d68fe023a11d44] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:24:23.171 INFO 3009027 --- [ocp-schedules-16,441ec77ce3ecf300,f0649447fcdbc1f3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:24:23.175 INFO 3009027 --- [ocp-schedules-14,16b768c869ca1190,d5d68fe023a11d44] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:24:23.175 INFO 3009027 --- [ocp-schedules-14,16b768c869ca1190,d5d68fe023a11d44] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:24:23.177 INFO 3009027 --- [ocp-schedules-13,fea52acd7cdedf09,601b11400848330b] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2025-04-08 20:24:23.177 INFO 3009027 --- [ocp-schedules-13,fea52acd7cdedf09,601b11400848330b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 20:24:23.177 INFO 3009027 --- [ocp-schedules-13,fea52acd7cdedf09,601b11400848330b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 20:24:23.190 INFO 3009027 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 20:24:23.191 INFO 3009027 --- [ocp-schedules-22,cec4f0281ab4f645,86e607d6e8bc77fa] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2025-04-08 20:24:23.192 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2025-04-08 20:24:23.194 INFO 3009027 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2025-04-08 20:24:23.195 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:24:23.200 INFO 3009027 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 493eecb387654b6f90f5379d8459859c. 2025-04-08 20:24:23.200 INFO 3009027 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2025-04-08 20:24:23.202 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1M43.865296348S 2025-04-08 20:24:23.202 INFO 3009027 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:24:23.209 INFO 3009027 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-async-7,3d8a8df5f5fb87af,84365d91ca3c5b06] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [] 2025-04-08 20:24:23.209 INFO 3009027 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2025-04-08 20:24:23.209 INFO 3009027 --- [ocp-async-7,3d8a8df5f5fb87af,84365d91ca3c5b06] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [] 2025-04-08 20:24:23.215 ERROR 3009027 --- [ocp-async-8,57a9765952f82fcf,a5535ddc6f66ff84] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2025-04-08 20:24:23.216 INFO 3009027 --- [ocp-async-8,57a9765952f82fcf,a5535ddc6f66ff84] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2025-04-08 20:24:23.216 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:24:23.216 INFO 3009027 --- [ocp-async-8,57a9765952f82fcf,a5535ddc6f66ff84] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2025-04-08 20:24:23.218 INFO 3009027 --- [ocp-async-8,57a9765952f82fcf,a5535ddc6f66ff84] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2025-04-08 20:24:23.221 WARN 3009027 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2025-04-08 20:24:23.221 INFO 3009027 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2025-04-08 20:24:23.221 INFO 3009027 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2025-04-08 20:24:23.514 INFO 3009027 --- [ocp-async-10,1308668168866431,b5d69730000c18c7] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2025-04-08 20:24:23.616 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:24:23.620 INFO 3009027 --- [ocp-async-10,1308668168866431,b5d69730000c18c7] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=623 2025-04-08 20:24:23.621 INFO 3009027 --- [ocp-async-10,1308668168866431,b5d69730000c18c7] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2025-04-08 20:24:23.950 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.831958344S 2025-04-08 20:24:23.950 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:24:24.016 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:24:24.018 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:24:24.044 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:24:24.103 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:24:24.103 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:24:24.103 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:24:24.103 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:24:24.103 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:24:24.103 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:24:24.110 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:24:24.211 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] o.s.boot.SpringApplication : Started application in 1.101 seconds (JVM running for 105.667) 2025-04-08 20:24:24.212 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.101306707S 2025-04-08 20:24:24.569 INFO 3009027 --- [prometheus-minute-schedule-1,9406a3b2b0b92646,ad382f0efe59aa04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:24.593 INFO 3009027 --- [ocp-schedules-27,a7e97656df072568,be8d2c652550da0e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:24:24.601 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2025-04-08 20:24:24.601 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2025-04-08 20:24:24.605 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Completed initialization in 4 ms 2025-04-08 20:24:24.795 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,2c9412e9c7ef20f9,ed45f45b01d3cacb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=2c9412e9c7ef20f9, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.799 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,2c9412e9c7ef20f9,ed45f45b01d3cacb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=2c9412e9c7ef20f9, duration=162 ms] 2025-04-08 20:24:24.878 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,bd44fea3828824bb,1c195c02f33286dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=bd44fea3828824bb, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.878 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,bd44fea3828824bb,1c195c02f33286dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=bd44fea3828824bb, duration=4 ms] 2025-04-08 20:24:24.887 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,2d965c2b6814225a,e5e730ef85ebabf1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=2d965c2b6814225a, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.887 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,2d965c2b6814225a,e5e730ef85ebabf1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=2d965c2b6814225a, duration=3 ms] 2025-04-08 20:24:24.893 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4ed6283bf5d884ba,c425117d82eac07a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=4ed6283bf5d884ba, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.893 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4ed6283bf5d884ba,c425117d82eac07a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=4ed6283bf5d884ba, duration=2 ms] 2025-04-08 20:24:24.900 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,ec1bbdd7e033af44,73e9da88b943f5c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=ec1bbdd7e033af44, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.900 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,ec1bbdd7e033af44,73e9da88b943f5c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=ec1bbdd7e033af44, duration=2 ms] 2025-04-08 20:24:24.907 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,181c01c41608d0f5,614c0464ceca5b9c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=181c01c41608d0f5, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.907 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,181c01c41608d0f5,614c0464ceca5b9c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=181c01c41608d0f5, duration=2 ms] 2025-04-08 20:24:24.914 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,5bd4b7e3d979eaa3,a911a014f8aef1ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=5bd4b7e3d979eaa3, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.914 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,5bd4b7e3d979eaa3,a911a014f8aef1ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=5bd4b7e3d979eaa3, duration=2 ms] 2025-04-08 20:24:24.920 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,41f8e7ba8dee51f8,b9ab4ccb102f995e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=41f8e7ba8dee51f8, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.920 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,41f8e7ba8dee51f8,b9ab4ccb102f995e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=41f8e7ba8dee51f8, duration=1 ms] 2025-04-08 20:24:24.926 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,bc62498f6780020a,9c9fd653ea3ab7a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=bc62498f6780020a, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.926 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,bc62498f6780020a,9c9fd653ea3ab7a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=bc62498f6780020a, duration=2 ms] 2025-04-08 20:24:24.931 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,333b3c26d32ce7fa,e863ebe79573c8d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=333b3c26d32ce7fa, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:24.931 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,333b3c26d32ce7fa,e863ebe79573c8d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=333b3c26d32ce7fa, duration=1 ms] 2025-04-08 20:24:25.072 WARN 3009027 --- [ocp-async-6,7b8c603c4e9b38cc,95c6fe6e8d199757] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:25.074 INFO 3009027 --- [ocp-async-1,f62e0358fae1a2ee,5b8303c8020ad9dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:25.979 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-04-08 20:24:26.116 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/software-packages?null, client=192.168.1.61, traceId=996136926ad2012a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage(MultipartFile), args=] 2025-04-08 20:24:26.133 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:26.134 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:26.142 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=100, bucket=sys-package, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, objectId=8 2025-04-08 20:24:27.073 WARN 3009027 --- [ocp-async-7,4480979675ad938e,a2ce246527bf5b04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:27.074 INFO 3009027 --- [ocp-async-10,ad80cbad65c41271,1dbe6503343345f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:27.139 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:24:27.569 INFO 3009027 --- [prometheus-minute-schedule-1,858e102be3d72d42,486db16c3a612a19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:29.073 WARN 3009027 --- [ocp-async-2,e5713d7abed1ce0f,b072e29f0641f625] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:29.074 INFO 3009027 --- [ocp-async-4,e7a0c7b2c8da5c67,b46b526605da8828] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:29.134 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:29.136 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:30.569 INFO 3009027 --- [prometheus-minute-schedule-1,01d4474c34649dac,93a5137aafd9ebb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:31.073 WARN 3009027 --- [ocp-async-6,52b546491c119286,2ee9324f9f9b226f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:31.074 INFO 3009027 --- [ocp-async-3,52930e9de4190ba8,ce6ef9be4aaa478e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:32.136 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:32.137 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:33.071 INFO 3009027 --- [ocp-async-7,46a4bec8e17cde6f,68fb4d0065551c85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:24:33.072 INFO 3009027 --- [ocp-schedules-28,f77aaf97c0fe60d9,397bbef02bb8243c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:24:33.073 WARN 3009027 --- [ocp-async-8,5a4a667b9c8e557b,567dc3706c3bdb66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:33.074 INFO 3009027 --- [ocp-async-2,bd9e5f2d2edf9f8e,f5511d14eedec1a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:33.197 INFO 3009027 --- [ocp-schedules-1,2a58c3b381597eb6,55d020fba7210449] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:24:33.199 INFO 3009027 --- [ocp-schedules-1,2a58c3b381597eb6,55d020fba7210449] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:24:33.569 INFO 3009027 --- [prometheus-minute-schedule-1,3b18b044bef95089,8aa1675b43d9e9c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:35.073 WARN 3009027 --- [ocp-async-10,476c66afe7d3982a,017f2050c17ac01d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:35.074 INFO 3009027 --- [ocp-async-6,c0fd5d44ceed02d3,33846933b5a9c390] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:35.137 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:35.138 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:36.569 INFO 3009027 --- [prometheus-minute-schedule-1,614cf1e065500219,47ffa74dc129384c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:37.073 WARN 3009027 --- [ocp-async-9,6cc46570936f8a36,924e3b56ae942576] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:37.075 INFO 3009027 --- [ocp-async-8,c4c357fa29e28c3e,2c3c3b1ff1f08f29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:37.591 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=8, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:24:37.592 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.o.s.s.SoftwarePackageServiceImpl : Save software object, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, objectId=8 2025-04-08 20:24:37.604 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.o.s.s.SoftwarePackageServiceImpl : Save software success, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, objectId=8, softwareId=6 2025-04-08 20:24:37.606 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.o.s.s.SoftwarePackageServiceImpl : Skip check package oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm signatures. 2025-04-08 20:24:37.608 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/software-packages client=192.168.1.61, traceId=996136926ad2012a, duration=12568 ms] 2025-04-08 20:24:37.610 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=UPLOAD_PACKAGE, eventType=SOFTWARE_PACKAGE_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage, resourceType=SOFTWARE_PACKAGE, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"file":"oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm"}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/software-packages, clientIp=192.168.1.61, serverIp=192.168.1.61, startTime=2025-04-08T20:24:26.115+08:00, endTime=2025-04-08T20:24:37.609+08:00, role=null, aliUid=null) 2025-04-08 20:24:37.613 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:24:37.616 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.o.service.event.RpmExtractListener : extract rpm start, rpmName=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, extractFiles=[./home/admin/oceanbase/etc/default_parameter.json, ./home/admin/oceanbase/etc/default_system_variable.json, ./home/admin/oceanbase/etc/oceanbase_upgrade_dep.yml] 2025-04-08 20:24:37.619 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] .o.o.s.o.i.LocalObjectStorageServiceImpl : Load object data, bucket=sys-package, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, objectId=8 2025-04-08 20:24:37.620 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.o.s.o.operator.LocalFileOperator : Bucket=sys-package, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, sha1=86eb4b858883f161075d8c41cd85388dfd964623 2025-04-08 20:24:37.655 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-10,996136926ad2012a,116585cc4a525e0f] o.h.p.entity.AbstractEntityPersister : HHH000502: The [objectMeta] property of the [com.oceanbase.ocp.core.metadb.repo.compute.softwarepackage.SoftwarePackageMetaEntity] entity was modified, but it won't be updated because the property is immutable. 2025-04-08 20:24:37.736 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,51e53003df6b40aa,60b3cc92506a0c91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=51e53003df6b40aa, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.736 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,51e53003df6b40aa,60b3cc92506a0c91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=51e53003df6b40aa, duration=2 ms] 2025-04-08 20:24:37.742 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,2652ec6645afcf56,68e700825c63db92] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=2652ec6645afcf56, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.742 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,2652ec6645afcf56,68e700825c63db92] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=2652ec6645afcf56, duration=1 ms] 2025-04-08 20:24:37.748 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,50eed0c24ea4bc65,43ba9b57e7d8e703] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=50eed0c24ea4bc65, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.748 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,50eed0c24ea4bc65,43ba9b57e7d8e703] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=50eed0c24ea4bc65, duration=2 ms] 2025-04-08 20:24:37.753 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,2ff21398a0968249,cc6f0d098fab526f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=2ff21398a0968249, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.753 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,2ff21398a0968249,cc6f0d098fab526f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=2ff21398a0968249, duration=1 ms] 2025-04-08 20:24:37.759 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5499d4f42fd1c520,65a3625872c31a4d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=5499d4f42fd1c520, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.759 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5499d4f42fd1c520,65a3625872c31a4d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=5499d4f42fd1c520, duration=3 ms] 2025-04-08 20:24:37.766 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8fc7e97f8eb17cc2,202b3229e37896d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=8fc7e97f8eb17cc2, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.766 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8fc7e97f8eb17cc2,202b3229e37896d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=8fc7e97f8eb17cc2, duration=3 ms] 2025-04-08 20:24:37.771 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,3ff93e12ead192bc,a30fdd21b1193916] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=3ff93e12ead192bc, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.771 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,3ff93e12ead192bc,a30fdd21b1193916] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=3ff93e12ead192bc, duration=1 ms] 2025-04-08 20:24:37.777 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,707ed4d187e65617,5ee156c5d9523f7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=707ed4d187e65617, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.777 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,707ed4d187e65617,5ee156c5d9523f7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=707ed4d187e65617, duration=1 ms] 2025-04-08 20:24:37.782 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,8f7b0463ffee803c,7fb61a86bfc71a41] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=8f7b0463ffee803c, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.782 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,8f7b0463ffee803c,7fb61a86bfc71a41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=8f7b0463ffee803c, duration=1 ms] 2025-04-08 20:24:37.787 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,de6f9c7ad4c6903b,b81734125be8f146] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=de6f9c7ad4c6903b, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.787 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,de6f9c7ad4c6903b,b81734125be8f146] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=de6f9c7ad4c6903b, duration=2 ms] 2025-04-08 20:24:37.793 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,abb6526abf213a35,388728c2f796b01d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=abb6526abf213a35, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:24:37.793 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,abb6526abf213a35,388728c2f796b01d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=abb6526abf213a35, duration=1 ms] 2025-04-08 20:24:38.138 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:38.140 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:38.201 INFO 3009027 --- [part-initial-worker,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 20:24:38.202 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 20:24:38.206 INFO 3009027 --- [part-initial-worker,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_meta, username=root, address=192.168.1.61, port=2881, database=meta_database) 2025-04-08 20:24:38.206 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_meta, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2025-04-08 20:24:38.217 INFO 3009027 --- [obsdk-init-thread-1,278f3715d78ffa41,6db082253dec4b2f] com.alibaba.druid.pool.DruidDataSource : {dataSource-3,ob-connector-datasource_direct_ocp_meta_root_192.168.1.61_2881} inited 2025-04-08 20:24:38.217 INFO 3009027 --- [obsdk-init-thread-1,278f3715d78ffa41,6db082253dec4b2f] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_meta, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2025-04-08 20:24:38.217 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 20:24:38.229 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.229 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:24:38.238 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.238 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:24:38.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 20:24:38.251 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1743523200 2025-04-08 20:24:38.253 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.254 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1743523200 values less than (1743523200)) 2025-04-08 20:24:38.481 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.482 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:38.686 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:38.688 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1743609600 2025-04-08 20:24:38.689 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.689 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1743609600 values less than (1743609600)) 2025-04-08 20:24:38.812 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.813 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:38.850 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:38.852 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1743696000 2025-04-08 20:24:38.853 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.853 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1743696000 values less than (1743696000)) 2025-04-08 20:24:38.937 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.938 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:38.961 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:38.963 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1743782400 2025-04-08 20:24:38.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:38.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1743782400 values less than (1743782400)) 2025-04-08 20:24:39.074 WARN 3009027 --- [ocp-async-8,a70f05f9bb927c6f,f4ed68d947fa9388] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:39.075 INFO 3009027 --- [ocp-async-7,ab1e5f9adbf942ca,d2ea7a0f3ee5b765] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:39.083 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:39.084 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:39.107 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:39.107 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1743868800 2025-04-08 20:24:39.108 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:39.108 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1743868800 values less than (1743868800)) 2025-04-08 20:24:39.480 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:39.481 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:39.501 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:39.502 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1743955200 2025-04-08 20:24:39.502 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:39.503 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1743955200 values less than (1743955200)) 2025-04-08 20:24:39.569 INFO 3009027 --- [prometheus-minute-schedule-1,0dafa76e48acf65f,d854485ac546b476] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:39.747 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:39.749 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:39.768 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:39.769 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1744041600 2025-04-08 20:24:39.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:39.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1744041600 values less than (1744041600)) 2025-04-08 20:24:40.003 INFO 3009027 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:24:40.003 INFO 3009027 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:24:40.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:40.082 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:40.103 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:40.104 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1744128000 2025-04-08 20:24:40.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:40.105 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1744128000 values less than (1744128000)) 2025-04-08 20:24:40.388 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:40.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:40.408 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:40.409 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1744214400 2025-04-08 20:24:40.410 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:40.410 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1744214400 values less than (1744214400)) 2025-04-08 20:24:40.689 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:40.690 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:40.708 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:40.709 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=profile_credential_sharing, partition=p1744300800 2025-04-08 20:24:40.709 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:40.710 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `profile_credential_sharing` add partition(partition p1744300800 values less than (1744300800)) 2025-04-08 20:24:41.074 WARN 3009027 --- [ocp-async-10,0e25c1d813981beb,3df91ef83a72070e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:41.075 INFO 3009027 --- [ocp-async-4,bfa423b1a93cfd9b,c525480411f538a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:41.140 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:41.141 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:41.175 INFO 3009027 --- [sql-rollup-daemon,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_monitor, username=root, address=192.168.1.61, port=2881, database=monitor_database) 2025-04-08 20:24:41.175 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2025-04-08 20:24:41.178 INFO 3009027 --- [obsdk-init-thread-2,67370bd598d90363,36fc302e28656839] com.alibaba.druid.pool.DruidDataSource : {dataSource-4,ob-connector-datasource_direct_ocp_monitor_root_192.168.1.61_2881} inited 2025-04-08 20:24:41.178 INFO 3009027 --- [obsdk-init-thread-2,67370bd598d90363,36fc302e28656839] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2025-04-08 20:24:41.179 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 20:24:41.180 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.181 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:24:41.183 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.183 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:24:41.184 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:24:41.186 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=2ms, period=120000ms 2025-04-08 20:24:41.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.189 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='profile_credential_sharing' and table_type='BASE TABLE' 2025-04-08 20:24:41.209 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=profile_credential_sharing 2025-04-08 20:24:41.211 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.212 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:24:41.212 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.212 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 20:24:41.267 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1743523200 2025-04-08 20:24:41.267 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.268 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1743523200 values less than (1743523200)) 2025-04-08 20:24:41.680 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.681 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:41.700 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:41.701 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1743609600 2025-04-08 20:24:41.701 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:41.702 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1743609600 values less than (1743609600)) 2025-04-08 20:24:42.134 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:42.136 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:42.158 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:42.159 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1743696000 2025-04-08 20:24:42.160 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:42.160 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1743696000 values less than (1743696000)) 2025-04-08 20:24:42.365 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:42.366 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:42.384 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:42.385 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1743782400 2025-04-08 20:24:42.385 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:42.386 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1743782400 values less than (1743782400)) 2025-04-08 20:24:42.569 INFO 3009027 --- [prometheus-minute-schedule-1,a6d3f8e669ebc42a,1c4888957eeccbeb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:42.674 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:42.675 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:42.692 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:42.693 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1743868800 2025-04-08 20:24:42.694 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:42.694 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1743868800 values less than (1743868800)) 2025-04-08 20:24:43.055 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.056 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:43.071 INFO 3009027 --- [ocp-async-9,fe841cab99c37092,7dab887d77b9df57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:24:43.073 INFO 3009027 --- [ocp-schedules-15,939714a28045bb9f,a4195b88d27029fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:24:43.073 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:43.074 WARN 3009027 --- [ocp-async-6,060b90e7c7076a69,f653b0ae798f8ffc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:43.074 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1743955200 2025-04-08 20:24:43.075 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.075 INFO 3009027 --- [ocp-async-3,f996ca45b32a8bca,99d0e3320af37e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:43.076 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1743955200 values less than (1743955200)) 2025-04-08 20:24:43.219 INFO 3009027 --- [ocp-schedules-10,0c37cb9ec2de4b0c,ee1d05e1276fe99b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:24:43.221 INFO 3009027 --- [ocp-schedules-10,0c37cb9ec2de4b0c,ee1d05e1276fe99b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:24:43.374 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.375 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:43.406 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:43.407 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1744041600 2025-04-08 20:24:43.407 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1744041600 values less than (1744041600)) 2025-04-08 20:24:43.647 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.648 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:43.668 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:43.669 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1744128000 2025-04-08 20:24:43.670 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.670 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1744128000 values less than (1744128000)) 2025-04-08 20:24:43.873 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.874 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:43.892 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:43.893 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1744214400 2025-04-08 20:24:43.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:43.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1744214400 values less than (1744214400)) 2025-04-08 20:24:44.141 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:44.142 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:44.175 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.176 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:44.196 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:44.196 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Metadb, table=ocp_cache_info, partition=p1744300800 2025-04-08 20:24:44.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_cache_info` add partition(partition p1744300800 values less than (1744300800)) 2025-04-08 20:24:44.349 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='meta_database' and table_name='ocp_cache_info' and table_type='BASE TABLE' 2025-04-08 20:24:44.369 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Metadb:table_name=ocp_cache_info 2025-04-08 20:24:44.370 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 20:24:44.370 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 20:24:44.370 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.371 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:24:44.371 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.372 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:24:44.373 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.373 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 20:24:44.389 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250406 2025-04-08 20:24:44.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.390 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:24:44.718 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.719 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:44.754 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:44.754 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250407 2025-04-08 20:24:44.755 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:44.756 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:24:45.073 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.074 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:45.074 WARN 3009027 --- [ocp-async-1,c2e69918f21bed77,8c07e579bbe10acd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:45.076 INFO 3009027 --- [ocp-async-7,2fb8b580e53e34b8,0f3504b2ac987682] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:45.091 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:45.092 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250408 2025-04-08 20:24:45.093 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.093 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:24:45.417 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.418 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:45.436 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:45.437 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250409 2025-04-08 20:24:45.437 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.438 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:24:45.569 INFO 3009027 --- [prometheus-minute-schedule-1,a3e308a90ae751a4,e0bfb37920840f63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:45.673 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.674 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:45.691 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:45.692 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250410 2025-04-08 20:24:45.692 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.693 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:24:45.896 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.897 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:45.915 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:45.915 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250411 2025-04-08 20:24:45.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:45.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:24:46.054 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.055 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:46.074 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:46.075 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250412 2025-04-08 20:24:46.075 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.076 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:24:46.315 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.316 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:46.335 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:46.336 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250413 2025-04-08 20:24:46.336 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:24:46.467 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.468 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:46.486 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:46.486 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250414 2025-04-08 20:24:46.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:24:46.728 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.729 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:46.748 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:46.749 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250415 2025-04-08 20:24:46.749 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.750 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:24:46.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.977 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:46.995 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:46.996 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_0, partition=P20250416 2025-04-08 20:24:46.996 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:46.997 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_0` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:24:47.074 WARN 3009027 --- [ocp-async-5,740c73815c0c2511,b7320cc0f7fe6431] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:47.076 INFO 3009027 --- [ocp-async-2,61c43258d81d73f3,f64d2153340093c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:47.142 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:47.143 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:47.283 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.284 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_0' and table_type='BASE TABLE' 2025-04-08 20:24:47.301 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_0 2025-04-08 20:24:47.302 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.302 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:24:47.303 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.303 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 20:24:47.313 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250331 2025-04-08 20:24:47.314 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.314 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:24:47.683 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.684 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:47.700 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:47.701 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250401 2025-04-08 20:24:47.701 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.702 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:24:47.929 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:47.947 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:47.948 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250402 2025-04-08 20:24:47.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:47.949 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:24:48.275 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:48.276 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:48.294 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:48.295 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250403 2025-04-08 20:24:48.295 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:48.296 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:24:48.403 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:48.404 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:48.421 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:48.422 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250404 2025-04-08 20:24:48.422 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:48.423 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:24:48.569 INFO 3009027 --- [prometheus-minute-schedule-1,59ab14a8df0307af,07fd2c4f67da8837] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:48.677 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:48.678 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:48.694 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:48.694 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250405 2025-04-08 20:24:48.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:48.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:24:49.024 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.025 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:49.042 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:49.043 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250406 2025-04-08 20:24:49.043 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.044 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:24:49.074 WARN 3009027 --- [ocp-async-4,2831bb392b59c5fb,9e77b8705f6529c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:49.076 INFO 3009027 --- [ocp-async-8,cdd3415696ca37a8,2811a277f90d9a0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:49.183 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.184 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:49.201 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:49.202 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250407 2025-04-08 20:24:49.202 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.202 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:24:49.491 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.492 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:49.509 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:49.510 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250408 2025-04-08 20:24:49.510 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.511 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:24:49.782 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.783 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:49.800 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:49.801 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250409 2025-04-08 20:24:49.802 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:49.802 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:24:50.143 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:50.144 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:50.325 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:50.326 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:50.344 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:50.344 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250410 2025-04-08 20:24:50.345 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:50.345 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:24:50.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:50.991 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:51.008 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:51.009 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250411 2025-04-08 20:24:51.010 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.010 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:24:51.075 WARN 3009027 --- [ocp-async-9,10445f1f3e21ca47,ebd09c1fd2419480] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:51.076 INFO 3009027 --- [ocp-async-3,ebdf9301a48e9eec,aef104a6d30cc01b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:51.521 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.522 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:51.541 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:51.542 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250412 2025-04-08 20:24:51.543 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.543 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:24:51.569 INFO 3009027 --- [prometheus-minute-schedule-1,ef81e1f6be85ddc9,aa6b74cd397dd16a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:51.728 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.729 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:51.746 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:51.747 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250413 2025-04-08 20:24:51.748 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.748 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:24:51.923 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.924 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:51.942 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:51.943 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250414 2025-04-08 20:24:51.943 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:51.944 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:24:52.079 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:52.097 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:52.098 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250415 2025-04-08 20:24:52.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:24:52.330 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.331 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:52.348 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:52.349 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_1, partition=P20250416 2025-04-08 20:24:52.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_1` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:24:52.556 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.557 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_1' and table_type='BASE TABLE' 2025-04-08 20:24:52.575 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_1 2025-04-08 20:24:52.576 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.576 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:24:52.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 20:24:52.592 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250324 2025-04-08 20:24:52.593 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.594 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:24:52.923 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.924 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:52.941 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:52.942 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250325 2025-04-08 20:24:52.943 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:52.943 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:24:53.057 INFO 3009027 --- [ocp-schedules-29,5bc4429bed0d8584,986ed9e4185f7120] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:24:53.057 INFO 3009027 --- [ocp-schedules-21,244c997e334bfc31,1fa8f84a460c70d3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:24:53.058 INFO 3009027 --- [ocp-schedules-1,f76729bddb879858,774cb35cbdd009e2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:24:53.058 INFO 3009027 --- [ocp-schedules-5,f691877e4d6faa91,7ef510ac35cfb207] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:24:53.062 INFO 3009027 --- [ocp-schedules-4,9e141b6c37799560,fb41dcc7168766d1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:24:53.071 INFO 3009027 --- [ocp-async-1,a55efa20ba56fa5b,fdacc28386b3c7b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:24:53.073 INFO 3009027 --- [ocp-schedules-13,854dcf547495ddf7,b2e8ca844e2cb3a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:24:53.076 WARN 3009027 --- [ocp-async-3,3355210669cc489e,edb82cd90d19887c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:53.077 INFO 3009027 --- [ocp-async-6,17608765dd015990,998ad4a27ccb38e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:53.101 INFO 3009027 --- [ocp-schedules-25,876d99a5262b7982,614c576356978a56] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:24:53.145 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:53.145 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:53.230 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:53.231 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:53.248 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:53.248 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250326 2025-04-08 20:24:53.249 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:53.249 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:24:53.341 INFO 3009027 --- [ocp-schedules-27,fde753a98a950d13,7460e88119246d07] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:24:53.342 INFO 3009027 --- [ocp-schedules-27,fde753a98a950d13,7460e88119246d07] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:24:53.634 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:53.635 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:53.651 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:53.652 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250327 2025-04-08 20:24:53.652 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:53.653 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:24:54.017 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:54.018 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:54.037 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:54.038 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250328 2025-04-08 20:24:54.039 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:54.039 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:24:54.360 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:54.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:54.380 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:54.381 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250329 2025-04-08 20:24:54.382 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:54.382 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:24:54.569 INFO 3009027 --- [prometheus-minute-schedule-1,3d306439d586386c,da408083ed749ab9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:54.727 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:54.728 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:54.745 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:54.746 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250330 2025-04-08 20:24:54.746 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:54.747 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:24:55.076 WARN 3009027 --- [ocp-async-2,0438a8fdcfe26442,ed68ac073952bede] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:55.077 INFO 3009027 --- [ocp-async-6,23a57af5f8760474,0e29c04b1fceb0fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:55.122 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:55.123 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:55.140 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:55.140 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250331 2025-04-08 20:24:55.141 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:55.141 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:24:55.493 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:55.494 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:55.514 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:55.515 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250401 2025-04-08 20:24:55.516 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:55.516 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:24:55.773 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:55.774 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:55.793 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:55.793 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250402 2025-04-08 20:24:55.794 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:55.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:24:56.025 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:56.026 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:56.044 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:56.045 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250403 2025-04-08 20:24:56.045 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:56.045 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:24:56.145 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:56.147 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:56.251 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.ocp.common.util.OcpProcessManager : CMD:[/bin/sh, -c, cd /tmp/oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm && rpm2cpio /home/root/software/sys-package/oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm | cpio -id ./home/admin/oceanbase/etc/default_parameter.json ./home/admin/oceanbase/etc/default_system_variable.json ./home/admin/oceanbase/etc/oceanbase_upgrade_dep.yml],PID:3014455,ENV: 2025-04-08 20:24:56.251 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.ocp.common.util.OcpProcessManager : result: 2025-04-08 20:24:56.367 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:56.368 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:56.387 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:56.388 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250404 2025-04-08 20:24:56.388 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:56.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:24:56.775 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:56.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:56.794 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:56.795 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250405 2025-04-08 20:24:56.796 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:56.796 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:24:57.076 WARN 3009027 --- [ocp-async-8,2e73986ac7a3aebb,360a26bf1a26a8bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:57.077 INFO 3009027 --- [ocp-async-6,e67ac2bf25d9e0ce,961c9a675fb8d70b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:57.298 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:57.299 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:57.318 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:57.319 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250406 2025-04-08 20:24:57.319 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:57.320 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:24:57.569 INFO 3009027 --- [prometheus-minute-schedule-1,e0ff92e43fa73162,db0032d32ebc8ad4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115100, interval=60 2025-04-08 20:24:57.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:57.741 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:57.758 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:57.758 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250407 2025-04-08 20:24:57.759 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:57.759 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:24:57.783 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] .o.o.o.i.p.ScenarioDefaultParameterSaver : saved ob scenario default parameter, rpmName=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, defaultParameterFile=/tmp/oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm/./home/admin/oceanbase/etc/default_parameter.json, defaultSystemVariableFile=/tmp/oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm/./home/admin/oceanbase/etc/default_system_variable.json 2025-04-08 20:24:57.802 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=oceanbase-upgrade-dep, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm.oceanbase_upgrade_dep.yml, objectId=9 2025-04-08 20:24:57.813 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=9, name=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm.oceanbase_upgrade_dep.yml 2025-04-08 20:24:57.813 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.o.o.i.cluster.ObUpgradeDepFileSaver : saved ob upgrade dep file, rpmName=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm, fileName=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm.oceanbase_upgrade_dep.yml 2025-04-08 20:24:57.825 INFO 3009027 --- [rpm-extract1,996136926ad2012a,8bbee99440354b71] c.o.o.service.event.RpmExtractListener : extract rpm end, rpmName=oceanbase-ce-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:24:57.974 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:57.975 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:57.992 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:57.993 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250408 2025-04-08 20:24:57.993 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:57.994 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:24:58.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:58.153 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:58.170 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:58.170 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250409 2025-04-08 20:24:58.171 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:58.172 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:24:58.510 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:58.511 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:58.529 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:58.530 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250410 2025-04-08 20:24:58.530 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:58.530 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:24:58.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:58.917 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:58.935 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:58.936 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250411 2025-04-08 20:24:58.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:58.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:24:59.076 WARN 3009027 --- [ocp-async-10,21298eaeeb5d72d9,70eef62931a15d6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:24:59.077 INFO 3009027 --- [ocp-async-7,f22396a14903a9b0,4aecd81b6bfb81b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:24:59.147 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:24:59.148 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:24:59.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:59.153 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:59.170 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:59.171 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250412 2025-04-08 20:24:59.172 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:59.172 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:24:59.473 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:59.474 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:59.492 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:59.493 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250413 2025-04-08 20:24:59.493 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:59.494 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:24:59.748 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:59.749 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:24:59.766 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:24:59.767 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250414 2025-04-08 20:24:59.767 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:24:59.768 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:00.195 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2025-04-08 20:25:00.0, currentDate=Tue Apr 08 20:25:00 CST 2025 2025-04-08 20:25:00.195 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=null, nextRunTime=2025-04-08 20:25:00.0} 2025-04-08 20:25:00.210 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 20:25:00.213 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:25:00.211+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:25:00.422 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:00.423 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:25:00.442 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:25:00.443 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250415 2025-04-08 20:25:00.443 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:00.444 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:00.519 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Start cache subtask classes 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.520 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeClusterHealthCheckerScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalPrimary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$LocalPrimary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalPrimary 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 20:25:00.521 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemoteStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$RemoteStandby 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePreScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Primary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$Primary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$Primary 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalPrimary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalPrimary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalPrimary 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 20:25:00.522 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=UpdateObClusterStatusTask, class=com.oceanbase.ocp.obops.internal.cluster.task.UpdateObClusterStatusTask, existClass=class com.oceanbase.ocp.service.task.business.host.UpdateObClusterStatusTask 2025-04-08 20:25:00.523 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=CreateResourcePoolTask, class=com.oceanbase.ocp.backup.internal.task.schedule.CreateResourcePoolTask, existClass=class com.oceanbase.ocp.service.task.business.tenant.CreateResourcePoolTask 2025-04-08 20:25:00.523 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Remove duplicated simple name cache, size=47 2025-04-08 20:25:00.523 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask cache initialized, simpleNameSize=565, classNameSize=623 2025-04-08 20:25:00.569 INFO 3009027 --- [prometheus-minute-schedule-1,5eca919f623ddba3,891fa76b050ac902] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:00.616 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2025-04-08 20:25:00.0, currentDate=Tue Apr 08 20:25:00 CST 2025 2025-04-08 20:25:00.616 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=null, nextRunTime=2025-04-08 20:25:00.0} 2025-04-08 20:25:00.619 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 20:25:00.644 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 20:25:00.644 INFO 3009027 --- [task-coordinator1,1308668168866431,5948525dc65d3525] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 20:25:00.703 INFO 3009027 --- [subtask-coordinator1,1308668168866431,6ff59a9aa7aec8dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1, operation:EXECUTE, state: RUNNING 2025-04-08 20:25:00.778 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:00.779 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:25:00.800 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:25:00.802 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat_2, partition=P20250416 2025-04-08 20:25:00.802 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:00.803 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat_2` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:00.878 INFO 3009027 --- [subtask-executor1,1308668168866431,5ee9fc25c836bbcf] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:25:00.880 INFO 3009027 --- [subtask-executor1,1308668168866431,5ee9fc25c836bbcf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1, context=Context{parallelIdx=-1, stringMap={task_instance_id=1, task_operation=execute, latest_execution_start_time=2025-04-08T20:25:00.692+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1}, listMap={}}, executor=192.168.1.61 2025-04-08 20:25:00.911 INFO 3009027 --- [subtask-executor1,1308668168866431,5ee9fc25c836bbcf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:25:01.051 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.052 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat_2' and table_type='BASE TABLE' 2025-04-08 20:25:01.073 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat_2 2025-04-08 20:25:01.075 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.075 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:01.076 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.076 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 20:25:01.077 WARN 3009027 --- [ocp-async-1,a054e417d4e51227,04ddb4ee6ccc4f10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:01.077 INFO 3009027 --- [ocp-async-2,b761a75b416ca884,49c19e84f7e1fc88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:01.087 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250309 2025-04-08 20:25:01.088 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.088 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250309 values less than (1741564800000000)) 2025-04-08 20:25:01.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.427 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:01.445 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:01.446 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250310 2025-04-08 20:25:01.446 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.447 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250310 values less than (1741651200000000)) 2025-04-08 20:25:01.660 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1, status:SUCCESS 2025-04-08 20:25:01.665 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1, status:SUCCESS 2025-04-08 20:25:01.676 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.677 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:01.694 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:01.694 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250311 2025-04-08 20:25:01.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:01.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250311 values less than (1741737600000000)) 2025-04-08 20:25:02.131 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.132 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:02.148 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:02.149 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:02.150 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:02.151 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250312 2025-04-08 20:25:02.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250312 values less than (1741824000000000)) 2025-04-08 20:25:02.257 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.258 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:02.275 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:02.276 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250313 2025-04-08 20:25:02.276 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.277 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250313 values less than (1741910400000000)) 2025-04-08 20:25:02.556 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.557 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:02.575 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:02.576 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250314 2025-04-08 20:25:02.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250314 values less than (1741996800000000)) 2025-04-08 20:25:02.807 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.808 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:02.827 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:02.828 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250315 2025-04-08 20:25:02.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:02.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250315 values less than (1742083200000000)) 2025-04-08 20:25:03.009 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.010 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:03.028 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:03.029 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250316 2025-04-08 20:25:03.029 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.030 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250316 values less than (1742169600000000)) 2025-04-08 20:25:03.071 INFO 3009027 --- [ocp-async-9,687700e9429664d5,2553f1ac88a171da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:25:03.073 INFO 3009027 --- [ocp-schedules-26,ea883c94f01a67e6,5a6059f17862d4a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:25:03.077 WARN 3009027 --- [ocp-async-5,103d53c6892ebc68,0b5dc53fbc0fb2ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:03.077 INFO 3009027 --- [ocp-async-3,3a8672938a382de3,75fd429d22fabc23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:03.286 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.287 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:03.305 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:03.306 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250317 2025-04-08 20:25:03.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.307 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250317 values less than (1742256000000000)) 2025-04-08 20:25:03.453 INFO 3009027 --- [ocp-schedules-25,89839a3eec9b2042,0684b6cdb16726c5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:25:03.455 INFO 3009027 --- [ocp-schedules-25,89839a3eec9b2042,0684b6cdb16726c5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:25:03.505 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.506 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:03.524 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:03.525 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250318 2025-04-08 20:25:03.525 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.526 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250318 values less than (1742342400000000)) 2025-04-08 20:25:03.569 INFO 3009027 --- [prometheus-minute-schedule-1,69a246741c079c18,aef0d713ae19cc1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:03.885 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.886 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:03.903 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:03.903 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250319 2025-04-08 20:25:03.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:03.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250319 values less than (1742428800000000)) 2025-04-08 20:25:04.083 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.084 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:04.102 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:04.103 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250320 2025-04-08 20:25:04.103 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250320 values less than (1742515200000000)) 2025-04-08 20:25:04.344 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.346 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:04.363 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:04.363 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250321 2025-04-08 20:25:04.364 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.364 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250321 values less than (1742601600000000)) 2025-04-08 20:25:04.593 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.594 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:04.613 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:04.614 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250322 2025-04-08 20:25:04.614 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.615 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250322 values less than (1742688000000000)) 2025-04-08 20:25:04.791 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.792 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:04.810 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:04.811 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250323 2025-04-08 20:25:04.811 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:04.812 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250323 values less than (1742774400000000)) 2025-04-08 20:25:05.077 WARN 3009027 --- [ocp-async-3,181129ba8a9e9332,bf93c71573e13e7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:05.078 INFO 3009027 --- [ocp-async-4,d88c176f9b0f3486,01034f39d4f2b419] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:05.149 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:05.150 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:05.294 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.295 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:05.313 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:05.313 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250324 2025-04-08 20:25:05.314 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.314 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:25:05.515 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.516 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:05.532 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:05.533 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250325 2025-04-08 20:25:05.533 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:25:05.782 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.783 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:05.799 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:05.800 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250326 2025-04-08 20:25:05.800 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.801 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:25:05.942 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.943 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:05.960 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:05.961 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250327 2025-04-08 20:25:05.962 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:05.962 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:25:06.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.198 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:06.215 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:06.215 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250328 2025-04-08 20:25:06.216 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.216 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:25:06.551 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.552 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:06.568 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:06.568 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250329 2025-04-08 20:25:06.569 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.569 INFO 3009027 --- [prometheus-minute-schedule-1,7f764897a69a1393,1f4062c6f0a27ff2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:06.569 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:25:06.715 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:06.733 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:06.733 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250330 2025-04-08 20:25:06.734 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.734 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:25:06.953 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.954 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:06.971 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:06.972 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250331 2025-04-08 20:25:06.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:06.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:25:07.077 WARN 3009027 --- [ocp-async-7,86142fe79410d7de,429eaec3416f85d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:07.078 INFO 3009027 --- [ocp-async-4,1bbb8d428059241b,96079625ccea09de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:07.143 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:07.160 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:07.161 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250401 2025-04-08 20:25:07.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.162 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:25:07.282 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.283 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:07.304 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:07.304 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250402 2025-04-08 20:25:07.305 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.305 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:25:07.504 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.505 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:07.523 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:07.524 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250403 2025-04-08 20:25:07.524 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.525 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:07.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.620 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:07.637 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:07.637 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250404 2025-04-08 20:25:07.638 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.638 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:07.810 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.811 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:07.828 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:07.829 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250405 2025-04-08 20:25:07.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:07.830 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:08.150 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:08.151 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:08.507 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:08.508 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:08.525 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:08.526 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250406 2025-04-08 20:25:08.526 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:08.527 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:08.881 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:08.882 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:08.900 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:08.900 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250407 2025-04-08 20:25:08.901 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:08.901 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:09.077 WARN 3009027 --- [ocp-async-4,100e5ad570cf29f7,2d6a58e5ccb15f1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:09.078 INFO 3009027 --- [ocp-async-8,4d84b3b2164e7e9b,fe9dd069e1d267d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:09.147 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:09.148 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:09.165 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:09.166 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250408 2025-04-08 20:25:09.167 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:09.167 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:09.391 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:09.392 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:09.408 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:09.409 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250409 2025-04-08 20:25:09.409 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:09.410 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:09.569 INFO 3009027 --- [prometheus-minute-schedule-1,b8d83d2a8a504efe,5704e811a0200ea7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:09.863 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:09.864 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:09.881 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:09.882 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250410 2025-04-08 20:25:09.883 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:09.883 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:10.002 INFO 3009027 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:25:10.002 INFO 3009027 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:25:10.125 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.126 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:10.139 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 0, ruleSize = 0. 2025-04-08 20:25:10.143 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:10.144 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250411 2025-04-08 20:25:10.145 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.145 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:10.281 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.282 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:10.300 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:10.301 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250412 2025-04-08 20:25:10.301 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.302 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:10.520 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.522 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:10.544 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:10.545 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250413 2025-04-08 20:25:10.545 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.546 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:10.874 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.875 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:10.893 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:10.894 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250414 2025-04-08 20:25:10.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:10.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:11.077 WARN 3009027 --- [ocp-async-3,ab157e2c0a8f17d6,227f83aba33850c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:11.078 INFO 3009027 --- [ocp-async-1,cd45d5cd84bd92ae,de6ab8161185f000] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:11.151 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:11.152 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:11.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.189 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:11.207 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:11.208 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250415 2025-04-08 20:25:11.208 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.209 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:11.413 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.414 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:11.431 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:11.432 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_stat, partition=P20250416 2025-04-08 20:25:11.432 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.433 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_stat` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:11.954 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.955 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_stat' and table_type='BASE TABLE' 2025-04-08 20:25:11.972 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_stat 2025-04-08 20:25:11.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:11.974 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.974 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 20:25:11.979 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250324 2025-04-08 20:25:11.979 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:11.980 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:25:12.347 ERROR 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing SQL_INDEX_DIAGNOSER org.springframework.dao.TransientDataAccessResourceException: ### Error updating database. Cause: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value ; (conn=3221489550) Table has no partition for value; nested exception is java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy354.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy585.save(Unknown Source) at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.save(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:25) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:48) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$b61f7705.getNextExecution() at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 43 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 44 common frames omitted 2025-04-08 20:25:12.362 ERROR 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing SQL_PERF_DIAGNOSER org.springframework.dao.TransientDataAccessResourceException: ### Error updating database. Cause: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value ; (conn=3221489550) Table has no partition for value; nested exception is java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy354.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy585.save(Unknown Source) at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.save(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:25) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:48) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$b61f7705.getNextExecution() at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 43 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 44 common frames omitted 2025-04-08 20:25:12.378 ERROR 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Error in executing PLAN_CHANGE_DETECTOR org.springframework.dao.TransientDataAccessResourceException: ### Error updating database. Cause: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value ### The error may exist in com/oceanbase/ocp/perf/sql/diag/monitordb/dao/OcpPerfSqlDiagnosisTaskDao.java (best guess) ### The error may involve com.oceanbase.ocp.perf.sql.diag.monitordb.dao.OcpPerfSqlDiagnosisTaskDao.save-Inline ### The error occurred while setting parameters ### SQL: REPLACE INTO ocp_perf_sql_diagnosis_task (task_name, begin_interval_time, end_interval_time, executor, status) VALUES (?, ?, ?, ?, ?) ### Cause: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value ; (conn=3221489550) Table has no partition for value; nested exception is java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:70) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:82) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:88) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440) at com.sun.proxy.$Proxy354.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62) at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85) at com.sun.proxy.$Proxy585.save(Unknown Source) at com.oceanbase.ocp.perf.sql.diag.internal.access.OcpPerfSqlDiagnosisTaskDbAccessImpl.save(OcpPerfSqlDiagnosisTaskDbAccessImpl.java:25) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService.getNextExecution(OcpPerfSqlDiagnosisTaskService.java:48) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$FastClassBySpringCGLIB$$a1a13d9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.perf.sql.diag.internal.OcpPerfSqlDiagnosisTaskService$$EnhancerBySpringCGLIB$$b61f7705.getNextExecution() at com.oceanbase.ocp.perf.sql.diag.internal.AbstractDiagnoseService.lambda$init$1(AbstractDiagnoseService.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLTransientConnectionException: (conn=3221489550) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426) ... 28 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 43 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 44 common frames omitted 2025-04-08 20:25:12.492 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:12.493 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:12.513 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:12.514 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250325 2025-04-08 20:25:12.514 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:12.515 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:25:12.569 INFO 3009027 --- [prometheus-minute-schedule-1,da21bd2935a2e5f0,d38d2ef019a9d095] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:12.570 INFO 3009027 --- [prometheus-cache-monitor1,b6257d122fa3693f,dfc174a089bc7554] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:25:12.570 INFO 3009027 --- [prometheus-cache-monitor1,b6257d122fa3693f,dfc174a089bc7554] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 20:25:12.741 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:12.742 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:12.759 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:12.760 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250326 2025-04-08 20:25:12.760 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:12.761 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:25:13.072 INFO 3009027 --- [ocp-async-1,6f5becadf23c1c3d,2b088e2a8919547f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:25:13.073 INFO 3009027 --- [ocp-schedules-10,077ddd502eff564b,276d2352a92cbf20] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:25:13.078 WARN 3009027 --- [ocp-async-7,ec8185bada25c53e,4f66731a6e2db82d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:13.078 INFO 3009027 --- [ocp-async-2,73db87114acd04e6,7f8bf802952a2063] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:13.329 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:13.330 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:13.347 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:13.348 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250327 2025-04-08 20:25:13.349 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:13.349 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:25:13.588 INFO 3009027 --- [ocp-schedules-27,8d4a4c6e17f58036,1834efdc8af51ebe] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:25:13.589 INFO 3009027 --- [ocp-schedules-27,8d4a4c6e17f58036,1834efdc8af51ebe] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:25:13.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:13.937 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:13.955 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:13.955 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250328 2025-04-08 20:25:13.956 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:13.956 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:25:14.146 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:14.147 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:14.152 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:14.153 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:14.163 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:14.164 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250329 2025-04-08 20:25:14.165 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:14.165 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:25:14.479 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:14.480 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:14.500 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:14.501 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250330 2025-04-08 20:25:14.502 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:14.502 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:25:14.824 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:14.825 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:14.842 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:14.842 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250331 2025-04-08 20:25:14.843 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:14.843 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:25:15.021 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:15.023 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:15.039 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:15.040 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250401 2025-04-08 20:25:15.040 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:15.041 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:25:15.078 WARN 3009027 --- [ocp-async-6,e6e7ca5b9bb8578c,9b19815b6ae49dc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:15.078 INFO 3009027 --- [ocp-async-2,bcfffbc3b30f0172,ad53327805556fe1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:15.532 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:15.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:15.551 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:15.552 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250402 2025-04-08 20:25:15.552 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:15.553 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:25:15.569 INFO 3009027 --- [prometheus-minute-schedule-1,5b6b8c20718bc5c8,a3954b72006d1131] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:15.851 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:15.852 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:15.871 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:15.872 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250403 2025-04-08 20:25:15.872 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:15.873 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:16.353 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:16.353 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:16.372 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:16.373 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250404 2025-04-08 20:25:16.374 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:16.374 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:16.987 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:16.987 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:17.005 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:17.006 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250405 2025-04-08 20:25:17.007 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:17.007 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:17.078 WARN 3009027 --- [ocp-async-9,269e14e33a0762aa,607bed696eb899e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:17.078 INFO 3009027 --- [ocp-async-8,9feab61ec8d07e8a,2a88705b541ba917] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:17.153 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:17.154 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:17.318 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:17.319 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:17.336 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:17.336 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250406 2025-04-08 20:25:17.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:17.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:17.903 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:17.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:17.924 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:17.925 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250407 2025-04-08 20:25:17.925 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:17.926 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:18.103 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:18.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:18.121 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:18.122 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250408 2025-04-08 20:25:18.122 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:18.123 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:18.516 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:18.517 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:18.534 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:18.534 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250409 2025-04-08 20:25:18.535 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:18.535 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:18.569 INFO 3009027 --- [prometheus-minute-schedule-1,cc3c85a7e182ff78,c1471cf2bb6669e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:18.844 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:18.845 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:18.863 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:18.864 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250410 2025-04-08 20:25:18.864 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:18.865 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:19.078 WARN 3009027 --- [ocp-async-5,39c6096b688ebcce,6269dbf8827e1f12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:19.079 INFO 3009027 --- [ocp-async-4,6f1105d464c41978,a6e9df4f8e5f9a6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:19.114 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:19.115 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:19.133 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:19.134 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250411 2025-04-08 20:25:19.134 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:19.134 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:19.432 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:19.433 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:19.451 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:19.452 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250412 2025-04-08 20:25:19.453 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:19.453 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:19.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:19.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:19.912 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:19.912 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250413 2025-04-08 20:25:19.913 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:19.913 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:20.154 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:20.155 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:20.428 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:20.429 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:20.447 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:20.448 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250414 2025-04-08 20:25:20.448 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:20.448 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:20.881 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:20.882 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:20.900 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:20.901 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250415 2025-04-08 20:25:20.901 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:20.902 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:21.078 WARN 3009027 --- [ocp-async-6,7a5d097da461a49d,a4f99c025a197e35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:21.079 INFO 3009027 --- [ocp-async-3,27f8937a4a3fabcf,40406e3353c171a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:21.388 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:21.407 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:21.407 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sqltext, partition=P20250416 2025-04-08 20:25:21.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sqltext` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:21.569 INFO 3009027 --- [prometheus-minute-schedule-1,616e2302756cfb85,c4e800865aac3821] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:21.641 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.642 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sqltext' and table_type='BASE TABLE' 2025-04-08 20:25:21.661 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sqltext 2025-04-08 20:25:21.662 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.662 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:21.663 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.664 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 20:25:21.667 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250324 2025-04-08 20:25:21.668 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.668 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:25:21.885 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.886 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:21.903 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:21.904 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250325 2025-04-08 20:25:21.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:21.905 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:25:22.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:22.145 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:22.161 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:22.162 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250326 2025-04-08 20:25:22.163 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:22.163 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:25:22.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:22.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:22.480 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:22.480 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250327 2025-04-08 20:25:22.481 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:22.481 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:25:22.818 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:22.819 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:22.836 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:22.836 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250328 2025-04-08 20:25:22.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:22.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:25:23.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.039 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:23.051 INFO 3009027 --- [ocp-schedules-6,e0c573139943fdc1,4a5b9b811bf597d6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:25:23.051+08:00, clusters=[] 2025-04-08 20:25:23.054 INFO 3009027 --- [ocp-schedules-7,e4d87429978587f4,8f7e1a05a3ca5b2c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:25:23.054+08:00, clusters=[] 2025-04-08 20:25:23.057 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:23.057 INFO 3009027 --- [ocp-schedules-26,e11be3a222c4eafa,78395df8107a1936] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:25:23.057 INFO 3009027 --- [ocp-schedules-20,5b360051565d4a1d,aa1fb032bf8a8553] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:25:23.057 INFO 3009027 --- [ocp-schedules-27,a6b187df02c4f853,df5ca8bda7dc95bd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:25:23.058 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250329 2025-04-08 20:25:23.058 INFO 3009027 --- [ocp-schedules-22,f522b4831dc60592,3199037cff502744] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:25:23.058 INFO 3009027 --- [ocp-schedules-5,bf26a586a13219c1,5d930ae330ad69b9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:25:23.058 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.059 INFO 3009027 --- [ocp-schedules-27,f62235a2f9a462cf,b9d8ca5703ce1e0b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:25:23.059 INFO 3009027 --- [ocp-schedules-24,1106a0b722670c0f,db49c7c820ae3ee8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:25:23.059 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:25:23.064 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.ocp.alarm.core.task.AlarmDetectTask : refresh host alarm wrapper 2025-04-08 20:25:23.068 INFO 3009027 --- [ocp-schedules-26,e11be3a222c4eafa,78395df8107a1936] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:25:23.069 INFO 3009027 --- [ocp-schedules-11,d77c33186476e980,ec5d5f3fc393973f] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:25:23.070 INFO 3009027 --- [ocp-schedules-15,be8094f1571b8261,365caf9971dcd087] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:25:23.070 INFO 3009027 --- [ocp-schedules-22,f522b4831dc60592,3199037cff502744] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:25:23.075 INFO 3009027 --- [ocp-schedules-11,d77c33186476e980,ec5d5f3fc393973f] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:25:23.075 INFO 3009027 --- [ocp-schedules-11,f4d31dea4985619c,c3d583a48d05364b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:25:23.075 INFO 3009027 --- [ocp-async-2,26fb90a24f9cedb0,9ff1b5bfd65d50bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:25:23.075 INFO 3009027 --- [ocp-schedules-11,e1e4e3623d720f8f,24ef0d915266b6b8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:25:23.078 INFO 3009027 --- [ocp-schedules-9,65ea06d161723c31,e913e098dfc11fbd] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 20:25:23.078 INFO 3009027 --- [ocp-schedules-9,65ea06d161723c31,e913e098dfc11fbd] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 20:25:23.078 WARN 3009027 --- [ocp-async-7,31e3f8d9434ef65d,f52fe68fa556f95e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:23.079 INFO 3009027 --- [ocp-async-8,71977cef4c65294b,522e442470c76ea0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:23.080 INFO 3009027 --- [ocp-schedules-21,d6469056f5204c2e,0d69c6ea8ae569bc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:25:23.086 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.ocp.alarm.core.task.AlarmDetectTask : refresh alarm target 2025-04-08 20:25:23.089 INFO 3009027 --- [ocp-schedules-6,d48621f88b197790,ec14d27d02a91483] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:25:23.093 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:25:23.102 INFO 3009027 --- [ocp-schedules-19,fd2cda8b2aa7c9c7,e11e4db3ae5251c1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:25:23.103 INFO 3009027 --- [ocp-schedules-5,16e6f27b55b661b9,392d4a8ad7916a55] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:25:23.109 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.a.service.TaskRepositoryService : fetchNeedExecute, tryAcquireTasksCount=1, fetchedCount=1 2025-04-08 20:25:23.113 INFO 3009027 --- [ocp-schedules-24,20dfb61d1583b300,1450dabef9aba84e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:25:23.113 INFO 3009027 --- [ocp-schedules-24,20dfb61d1583b300,1450dabef9aba84e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:25:23.117 INFO 3009027 --- [ocp-schedules-24,20dfb61d1583b300,1450dabef9aba84e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:25:23.117 INFO 3009027 --- [ocp-schedules-24,20dfb61d1583b300,1450dabef9aba84e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:25:23.118 INFO 3009027 --- [ocp-schedules-27,bbbb0b7efca1ebf8,bc64aee5f0ee21ba] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:25:23.118 INFO 3009027 --- [ocp-schedules-27,bbbb0b7efca1ebf8,bc64aee5f0ee21ba] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:25:23.135 INFO 3009027 --- [ocp-schedules-20,722d996b1835a99e,fed2336d12c930e5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:25:23.142 INFO 3009027 --- [ocp-schedules-8,5673df5c56ecd366,33cd1bb365352a39] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:25:23.142 INFO 3009027 --- [ocp-schedules-8,5673df5c56ecd366,33cd1bb365352a39] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:25:23.145 INFO 3009027 --- [ocp-schedules-20,722d996b1835a99e,fed2336d12c930e5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:25:23.145 INFO 3009027 --- [ocp-schedules-20,722d996b1835a99e,fed2336d12c930e5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:25:23.155 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:23.156 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:23.188 INFO 3009027 --- [ocp-schedules-10,ba9dd59fd6928b18,ff3bc01c2b360b3c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:25:23.188 INFO 3009027 --- [ocp-schedules-10,ba9dd59fd6928b18,ff3bc01c2b360b3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:25:23.191 INFO 3009027 --- [ocp-schedules-10,ba9dd59fd6928b18,ff3bc01c2b360b3c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:25:23.191 INFO 3009027 --- [ocp-schedules-10,ba9dd59fd6928b18,ff3bc01c2b360b3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:25:23.194 INFO 3009027 --- [ocp-schedules-10,ba9dd59fd6928b18,ff3bc01c2b360b3c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:25:23.194 INFO 3009027 --- [ocp-schedules-10,ba9dd59fd6928b18,ff3bc01c2b360b3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:25:23.229 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 20:25:23.236 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.237 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 20:25:23.237 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 20:25:23.237 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 20:25:23.238 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 20:25:23.240 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 20:25:23.241 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 20:25:23.261 INFO 3009027 --- [ocp-schedules-4,1765a86bf51bdae9,cba370e1e2d1df45] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets , current targets ,,,,,, 2025-04-08 20:25:23.262 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 20:25:23.263 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 20:25:23.263 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.263 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 20:25:23.263 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 20:25:23.263 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 20:25:23.264 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 20:25:23.265 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 20:25:23.266 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:25:23.266 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.266 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.266 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.267 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.267 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.267 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 20:25:23.267 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 20:25:23.268 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 20:25:23.268 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2025-04-08 20:25:23.268 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 20:25:23.268 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 20:25:23.268 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 20:25:23.269 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 20:25:23.270 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 20:25:23.271 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 20:25:23.272 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 20:25:23.273 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 20:25:23.274 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 20:25:23.275 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 20:25:23.276 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 20:25:23.277 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 20:25:23.278 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 20:25:23.279 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 20:25:23.280 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 20:25:23.281 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 20:25:23.282 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 20:25:23.283 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 20:25:23.284 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 20:25:23.285 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 20:25:23.286 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 20:25:23.287 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 20:25:23.288 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 20:25:23.289 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 20:25:23.290 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 20:25:23.291 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 20:25:23.292 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 20:25:23.293 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 20:25:23.294 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 20:25:23.295 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 20:25:23.296 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets 2025-04-08 20:25:23.297 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 20:25:23.297 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 20:25:23.297 INFO 3009027 --- [ocp-schedules-30,b5a53bcd8e80cac5,3f321c6ee03f2382] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 20:25:23.322 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.alarm.core.process.AlarmProcessor : AlarmProcessor init done, lastProcessTime=2025-04-08T20:25:23.174237+08:00 2025-04-08 20:25:23.323 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.alarm.core.task.AlarmProcessTask : AlarmProcessTask created 2025-04-08 20:25:23.323 INFO 3009027 --- [ocp-schedules-22,3d6a172493ac5749,4380e87bec347aef] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:25:23.323 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.alarm.utils.task.AlarmTaskManager : start task, taskInfo=TaskInfo(id=1, uuid=254bed2a-3c53-4ecf-b850-9c2422739b3a, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=EXECUTING, executorName=192.168.1.61:3009027, createAt=2025-04-08T12:21:34.832298Z, startAt=2025-04-08T12:25:23.094Z, endAt=null, heartbeatAt=2025-04-08T12:25:23.094Z, result=executing) 2025-04-08 20:25:23.324 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.alarm.utils.task.AlarmTaskManager : startNeedExecute, activeCount=0, tryStartCount=1, startedCount=1 2025-04-08 20:25:23.338 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:25:23.338 INFO 3009027 --- [ocp-schedules-14,84a40531e0a4abb4,2cfb2e138cf2dc86] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:25:23.342 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 20:25:23.342 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.342 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 20:25:23.342 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 20:25:23.343 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 20:25:23.344 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 20:25:23.344 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 20:25:23.345 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 20:25:23.345 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 20:25:23.346 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.347 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 20:25:23.348 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 20:25:23.349 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 20:25:23.350 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 20:25:23.351 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 20:25:23.352 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 20:25:23.353 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 20:25:23.354 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 20:25:23.355 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.356 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 20:25:23.357 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 20:25:23.358 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 20:25:23.359 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 20:25:23.360 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 20:25:23.361 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 20:25:23.362 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 20:25:23.363 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 20:25:23.364 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 20:25:23.364 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 20:25:23.364 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:23.364 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 20:25:23.364 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 20:25:23.364 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 20:25:23.365 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250330 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 20:25:23.365 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 20:25:23.365 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 20:25:23.366 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 20:25:23.366 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 20:25:23.367 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 20:25:23.368 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 20:25:23.369 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 20:25:23.370 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:25:23.371 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:25:23.372 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 20:25:23.372 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.372 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 20:25:23.372 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 20:25:23.373 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 20:25:23.374 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 20:25:23.375 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:25:23.374419+08:00, lastProcessTime=2025-04-08T20:25:23.174237+08:00 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 20:25:23.375 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 20:25:23.376 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 20:25:23.377 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 20:25:23.378 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 20:25:23.379 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 20:25:23.380 INFO 3009027 --- [ocp-schedules-16,d63997d1e2d15a8d,bee748206eb28e5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:25:23.417 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.418 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:23.441 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:23.442 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250331 2025-04-08 20:25:23.442 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.443 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:25:23.622 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.623 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:23.640 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:23.640 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250401 2025-04-08 20:25:23.645 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.645 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:25:23.649 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:25:23.374419+08:00 2025-04-08 20:25:23.700 INFO 3009027 --- [ocp-schedules-25,79b0a3ecc30d886b,ec64990994e2a0db] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:25:23.702 INFO 3009027 --- [ocp-schedules-25,79b0a3ecc30d886b,ec64990994e2a0db] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:25:23.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.810 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:23.828 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:23.829 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250402 2025-04-08 20:25:23.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:25:23.974 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.975 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:23.994 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:23.995 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250403 2025-04-08 20:25:23.996 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:23.996 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:24.165 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.165 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:24.185 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:24.185 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250404 2025-04-08 20:25:24.186 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.186 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:24.417 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.418 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:24.434 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:24.435 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250405 2025-04-08 20:25:24.435 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.435 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:24.569 INFO 3009027 --- [prometheus-minute-schedule-1,96182099b1a833c0,5e3da01a60af9156] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:24.604 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.605 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:24.622 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:24.623 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250406 2025-04-08 20:25:24.623 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.624 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:24.791 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.792 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:24.808 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:24.809 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250407 2025-04-08 20:25:24.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:24.810 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:25.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:25.052 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:25.053 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250408 2025-04-08 20:25:25.053 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.054 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:25.079 WARN 3009027 --- [ocp-async-2,bd0f4aa3242d6800,76dfa94905b5e2dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:25.079 INFO 3009027 --- [ocp-async-4,1fd76388a2d0c9c6,a681c1ea4727ad91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:25.319 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.320 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:25.336 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:25.337 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250409 2025-04-08 20:25:25.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:25.653 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.654 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:25.671 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:25.672 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250410 2025-04-08 20:25:25.673 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.673 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:25.791 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.791 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:25.808 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:25.808 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250411 2025-04-08 20:25:25.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:25.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:25.991 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:26.011 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:26.012 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250412 2025-04-08 20:25:26.012 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.013 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:26.156 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:26.157 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:26.374 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.375 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:26.393 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:26.394 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250413 2025-04-08 20:25:26.394 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.394 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:26.600 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.602 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:26.618 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:26.619 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250414 2025-04-08 20:25:26.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.620 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:26.794 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:26.812 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:26.813 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250415 2025-04-08 20:25:26.813 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:26.814 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:27.011 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.012 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:27.029 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:27.029 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_dynamic_sql, partition=P20250416 2025-04-08 20:25:27.030 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.030 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_dynamic_sql` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:27.079 WARN 3009027 --- [ocp-async-4,17ec0646965a5530,7773982ee01324c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:27.079 INFO 3009027 --- [ocp-async-10,5a45692702e674d2,93d6ef5371199993] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:27.139 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:25:27.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.162 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_dynamic_sql' and table_type='BASE TABLE' 2025-04-08 20:25:27.178 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_dynamic_sql 2025-04-08 20:25:27.179 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.180 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:27.180 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.181 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 20:25:27.184 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250324 2025-04-08 20:25:27.184 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.185 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:25:27.421 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.422 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:27.439 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:27.440 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250325 2025-04-08 20:25:27.440 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.440 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:25:27.569 INFO 3009027 --- [prometheus-minute-schedule-1,74cf5241c72c50f0,c0c6d5f40e9a1b3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:27.572 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.573 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:27.590 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:27.591 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250326 2025-04-08 20:25:27.591 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.592 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:25:27.985 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:27.986 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:28.002 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:28.003 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250327 2025-04-08 20:25:28.004 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:28.004 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:25:28.232 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:28.233 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:28.250 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:28.251 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250328 2025-04-08 20:25:28.251 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:28.252 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:25:28.796 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:28.797 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:28.814 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:28.815 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250329 2025-04-08 20:25:28.815 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:28.816 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:25:29.055 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.056 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:29.073 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:29.073 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250330 2025-04-08 20:25:29.074 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.074 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:25:29.079 WARN 3009027 --- [ocp-async-1,0294a6d19b50bcab,28ef13d8880116ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:29.079 INFO 3009027 --- [ocp-async-7,32741d56566af152,01f9ac84cf9c326f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:29.157 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:29.158 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:29.353 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.354 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:29.371 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:29.371 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250331 2025-04-08 20:25:29.372 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.372 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:25:29.431 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.432 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:29.449 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:29.450 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250401 2025-04-08 20:25:29.450 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.451 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:25:29.769 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.769 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:29.787 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:29.788 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250402 2025-04-08 20:25:29.788 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.788 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:25:29.885 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.886 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:29.903 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:29.904 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250403 2025-04-08 20:25:29.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:29.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:30.035 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.036 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.052 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.053 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250404 2025-04-08 20:25:30.053 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.054 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:30.201 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.202 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.221 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.222 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250405 2025-04-08 20:25:30.222 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:30.274 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.275 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.292 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.292 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250406 2025-04-08 20:25:30.293 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.293 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:30.420 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.421 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.437 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.438 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250407 2025-04-08 20:25:30.438 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.439 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:30.533 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.550 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.551 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250408 2025-04-08 20:25:30.551 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.551 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:30.569 INFO 3009027 --- [prometheus-minute-schedule-1,edfdd8220b3ab375,3cdabc3ab22fec1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:30.606 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.607 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.624 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.625 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250409 2025-04-08 20:25:30.625 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.626 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:30.778 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.779 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:30.798 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:30.799 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250410 2025-04-08 20:25:30.799 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:30.800 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:31.052 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.053 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:31.071 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:31.071 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250411 2025-04-08 20:25:31.072 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.072 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:31.079 WARN 3009027 --- [ocp-async-7,2086f22c952935ee,d8801f8cab107f72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:31.079 INFO 3009027 --- [ocp-async-8,00a21ad02661f928,e8828320fb406b24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:31.246 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.247 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:31.265 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:31.266 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250412 2025-04-08 20:25:31.266 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.267 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:31.835 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.836 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:31.856 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:31.857 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250413 2025-04-08 20:25:31.857 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.858 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:31.961 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.962 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:31.978 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:31.978 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250414 2025-04-08 20:25:31.979 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:31.979 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:32.116 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.117 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:32.134 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:32.134 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250415 2025-04-08 20:25:32.135 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.135 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:32.158 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:32.159 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:32.176 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.177 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:32.195 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:32.196 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_high_risk_sql, partition=P20250416 2025-04-08 20:25:32.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_high_risk_sql` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:32.276 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.277 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_high_risk_sql' and table_type='BASE TABLE' 2025-04-08 20:25:32.295 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_high_risk_sql 2025-04-08 20:25:32.296 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.296 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:32.297 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.297 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:32.302 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250403 2025-04-08 20:25:32.303 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.303 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:32.689 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.690 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:32.707 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:32.707 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250404 2025-04-08 20:25:32.708 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:32.708 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:33.012 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:33.013 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:33.032 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:33.032 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250405 2025-04-08 20:25:33.033 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:33.033 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:33.075 INFO 3009027 --- [ocp-schedules-9,5011c7490b650e0c,6065dc4cd1a9c7cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:25:33.075 INFO 3009027 --- [ocp-async-4,b5903a828a74938a,e5d57e5fa7fac256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:25:33.080 WARN 3009027 --- [ocp-async-5,50dc526a2b2835f9,02002cfd4dd891f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:33.080 INFO 3009027 --- [ocp-async-1,c4eb986f1fe4d38b,9877cd30da9b0e6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:33.179 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:25:33.178881+08:00, lastProcessTime=2025-04-08T20:25:23.374419+08:00 2025-04-08 20:25:33.434 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:25:33.178881+08:00 2025-04-08 20:25:33.446 INFO 3009027 --- [ocp-schedules-13,e8fd29847d1af51c,06ec6c94f5ca121d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:25:33.569 INFO 3009027 --- [prometheus-minute-schedule-1,9dc258e6242f2af5,0111c7c1791f41d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:33.609 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:33.610 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:33.629 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:33.630 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250406 2025-04-08 20:25:33.630 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:33.631 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:33.733 INFO 3009027 --- [ocp-schedules-18,b06d3ab0311c17c4,88ea000fdb7d2ff6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:25:33.734 INFO 3009027 --- [ocp-schedules-18,b06d3ab0311c17c4,88ea000fdb7d2ff6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:25:33.871 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:33.872 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:33.889 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:33.890 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250407 2025-04-08 20:25:33.890 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:33.890 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:34.122 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.123 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:34.139 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:34.140 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250408 2025-04-08 20:25:34.140 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.141 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:34.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.362 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:34.377 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:34.378 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250409 2025-04-08 20:25:34.378 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.379 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:34.676 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.676 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:34.693 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:34.694 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250410 2025-04-08 20:25:34.694 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:34.814 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.815 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:34.832 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:34.832 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250411 2025-04-08 20:25:34.833 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.833 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:34.962 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.963 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:34.984 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:34.984 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250412 2025-04-08 20:25:34.985 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:34.985 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:35.079 WARN 3009027 --- [ocp-async-7,473ac13e2af8f0a4,91fe6a9f290a0a5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:35.080 INFO 3009027 --- [ocp-async-1,11aa4a2cda1bf5e1,731410b8ca491b31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:35.159 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:35.159 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:35.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:35.198 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:35.217 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:35.217 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250413 2025-04-08 20:25:35.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:35.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:35.607 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:35.608 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:35.625 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:35.626 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250414 2025-04-08 20:25:35.627 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:35.627 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:36.046 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.047 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:36.064 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:36.065 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250415 2025-04-08 20:25:36.065 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.066 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:36.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.500 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:36.517 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:36.517 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0, partition=P20250416 2025-04-08 20:25:36.518 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.518 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_stat_0` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:36.569 INFO 3009027 --- [prometheus-minute-schedule-1,3201911a7c44537c,752b0dbb5e5c5110] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:36.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.949 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_stat_0' and table_type='BASE TABLE' 2025-04-08 20:25:36.966 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:25:36.967 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.967 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:36.968 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.968 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 20:25:36.973 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250403 2025-04-08 20:25:36.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:36.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:37.080 WARN 3009027 --- [ocp-async-1,081752a14869a3e4,70a578ccaf6d444d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:37.080 INFO 3009027 --- [ocp-async-8,8b427c32de10a385,1ef18e843d8e483a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:37.251 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:37.252 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:37.269 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:37.270 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250404 2025-04-08 20:25:37.270 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:37.270 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:37.641 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:37.642 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:37.662 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:37.663 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250405 2025-04-08 20:25:37.663 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:37.664 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:37.942 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:37.943 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:37.962 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:37.963 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250406 2025-04-08 20:25:37.963 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:37.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:38.160 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:38.160 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:38.673 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:38.674 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:38.692 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:38.692 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250407 2025-04-08 20:25:38.693 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:38.693 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:38.923 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:38.924 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:38.940 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:38.941 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250408 2025-04-08 20:25:38.941 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:38.941 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:39.080 WARN 3009027 --- [ocp-async-8,c95e69cbd26b0d4c,2f54856408a31fc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:39.080 INFO 3009027 --- [ocp-async-10,50e4504783741d87,1752968d86e34ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:39.148 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:39.149 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:39.165 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:39.166 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250409 2025-04-08 20:25:39.166 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:39.167 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:39.569 INFO 3009027 --- [prometheus-minute-schedule-1,acce3babcc52a8d8,8556a9987ebb511a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:39.615 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:39.616 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:39.635 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:39.636 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250410 2025-04-08 20:25:39.636 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:39.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:39.966 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:39.967 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:39.985 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:39.986 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250411 2025-04-08 20:25:39.986 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:39.987 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:40.001 INFO 3009027 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:25:40.002 INFO 3009027 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:25:40.271 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:40.273 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:40.289 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:40.290 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250412 2025-04-08 20:25:40.290 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:40.291 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:40.775 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:40.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:40.793 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:40.794 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250413 2025-04-08 20:25:40.794 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:40.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:41.014 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:41.016 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:41.033 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:41.033 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250414 2025-04-08 20:25:41.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:41.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:41.080 WARN 3009027 --- [ocp-async-10,0ec055ca4c4a118d,d562975403c74da3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:41.080 INFO 3009027 --- [ocp-async-7,f6fe7ceb63684cfc,3e919d6a41cb91eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:41.161 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:41.161 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:41.338 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:41.339 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:41.356 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:41.356 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250415 2025-04-08 20:25:41.357 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:41.357 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:41.720 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:41.721 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:41.739 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:41.739 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan, partition=P20250416 2025-04-08 20:25:41.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:41.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:42.120 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.121 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan' and table_type='BASE TABLE' 2025-04-08 20:25:42.138 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan 2025-04-08 20:25:42.139 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.139 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:42.139 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.140 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 20:25:42.144 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250403 2025-04-08 20:25:42.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.145 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:42.244 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.246 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:42.263 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:42.263 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250404 2025-04-08 20:25:42.264 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.264 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:42.496 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.497 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:42.517 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:42.518 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250405 2025-04-08 20:25:42.518 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.519 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:42.569 INFO 3009027 --- [prometheus-minute-schedule-1,b6e24c5db3951e4d,1c2714edc8b69c1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:42.612 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.613 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:42.630 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:42.631 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250406 2025-04-08 20:25:42.631 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.631 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:42.835 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.836 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:42.852 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:42.853 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250407 2025-04-08 20:25:42.853 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:42.854 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:43.012 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.013 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:43.030 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:43.031 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250408 2025-04-08 20:25:43.031 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.032 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:43.075 INFO 3009027 --- [ocp-schedules-23,07568214a907e843,300a459a4bc39322] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:25:43.075 INFO 3009027 --- [ocp-async-7,4bc5fef58136395a,b57765ba7f91fbbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:25:43.080 WARN 3009027 --- [ocp-async-1,5fac6d51156ddecc,201bb445acc48f29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:43.080 INFO 3009027 --- [ocp-async-1,89e5f5ea9bd0d311,54af67c99b29318d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:43.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.145 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:43.162 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:43.163 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250409 2025-04-08 20:25:43.163 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.164 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:43.179 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:25:43.178616+08:00, lastProcessTime=2025-04-08T20:25:33.178881+08:00 2025-04-08 20:25:43.339 INFO 3009027 --- [ocp-schedules-3,e517c51fd9f375b4,c11797f467570510] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:25:43.350 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:25:43.178616+08:00 2025-04-08 20:25:43.393 INFO 3009027 --- [ocp-schedules-3,e517c51fd9f375b4,c11797f467570510] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:25:43.394 INFO 3009027 --- [ocp-schedules-3,e517c51fd9f375b4,c11797f467570510] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:25:43.492 INFO 3009027 --- [ocp-schedules-6,3ee42eba2f1c15bd,d82c3765af99fa4e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:25:43.538 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.538 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:43.556 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:43.557 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250410 2025-04-08 20:25:43.557 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.558 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:43.733 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.734 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:43.752 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:43.753 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250411 2025-04-08 20:25:43.753 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.754 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:43.758 INFO 3009027 --- [ocp-schedules-13,4495aa5a879f0e94,7efc35bef5a749aa] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:25:43.759 INFO 3009027 --- [ocp-schedules-13,4495aa5a879f0e94,7efc35bef5a749aa] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:25:43.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.905 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:43.921 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:43.922 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250412 2025-04-08 20:25:43.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:43.923 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:44.162 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:44.162 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:44.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:44.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:44.367 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:44.368 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250413 2025-04-08 20:25:44.369 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:44.369 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:44.589 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:44.590 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:44.608 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:44.609 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250414 2025-04-08 20:25:44.609 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:44.610 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:45.080 WARN 3009027 --- [ocp-async-5,f5b2a6ed8512ed3c,1481adb24c2772fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:45.080 INFO 3009027 --- [ocp-async-9,e320ce2762c0545f,ac7f2d573140754e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:45.112 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.113 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:45.131 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:45.131 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250415 2025-04-08 20:25:45.132 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.132 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:45.467 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.468 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:45.486 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:45.486 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_plan_cache_plan_explain, partition=P20250416 2025-04-08 20:25:45.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_plan_cache_plan_explain` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:45.569 INFO 3009027 --- [prometheus-minute-schedule-1,71c05973293d7efc,0799bf0c80f80c68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:45.914 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.915 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_plan_cache_plan_explain' and table_type='BASE TABLE' 2025-04-08 20:25:45.933 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_plan_cache_plan_explain 2025-04-08 20:25:45.934 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.934 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:45.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.937 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 20:25:45.944 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250401 2025-04-08 20:25:45.944 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:45.944 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:25:46.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:46.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:46.367 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:46.368 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250402 2025-04-08 20:25:46.368 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:46.369 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:25:46.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:46.741 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:46.757 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:46.757 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250403 2025-04-08 20:25:46.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:46.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:47.019 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:47.020 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:47.036 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:47.037 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250404 2025-04-08 20:25:47.037 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:47.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:47.080 WARN 3009027 --- [ocp-async-9,852fa98eaa53932e,1c751c541eea48ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:47.081 INFO 3009027 --- [ocp-async-6,59aede348bf94090,e5d038cab786eafa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:47.163 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:47.163 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:47.480 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:47.481 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:47.497 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:47.498 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250405 2025-04-08 20:25:47.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:47.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:47.947 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:47.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:47.964 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:47.964 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250406 2025-04-08 20:25:47.965 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:47.965 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:48.383 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:48.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:48.400 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:48.401 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250407 2025-04-08 20:25:48.401 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:48.402 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:48.569 INFO 3009027 --- [prometheus-minute-schedule-1,589479a3bf94a151,28caefbbc3d25c52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:48.876 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:48.877 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:48.896 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:48.897 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250408 2025-04-08 20:25:48.897 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:48.897 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:49.081 WARN 3009027 --- [ocp-async-6,fdd2f468738ffb46,1085ebde01f6ceba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:49.081 INFO 3009027 --- [ocp-async-7,5c6523f3d0b5dc97,5ab9cf4d1f53c16d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:49.226 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:49.227 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:49.244 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:49.245 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250409 2025-04-08 20:25:49.245 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:49.246 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:49.658 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:49.659 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:49.677 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:49.677 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250410 2025-04-08 20:25:49.678 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:49.678 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:49.971 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:49.972 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:49.989 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:49.990 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250411 2025-04-08 20:25:49.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:49.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:50.163 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:50.164 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:50.443 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:50.444 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:50.460 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:50.461 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250412 2025-04-08 20:25:50.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:50.462 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:50.849 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:50.850 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:50.869 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:50.869 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250413 2025-04-08 20:25:50.870 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:50.870 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:51.081 WARN 3009027 --- [ocp-async-10,32e4f4503c6a4c26,35b4fbac66e3d120] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:51.081 INFO 3009027 --- [ocp-async-7,f59638e86a348801,9dd93a22c800886d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:51.134 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:51.135 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:51.151 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:51.152 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250414 2025-04-08 20:25:51.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:51.153 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:51.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:51.427 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:51.445 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:51.446 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250415 2025-04-08 20:25:51.446 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:51.447 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:51.569 INFO 3009027 --- [prometheus-minute-schedule-1,d4d64ded462b08f7,c399957b9da19244] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:51.697 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:51.698 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:51.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:51.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample, partition=P20250416 2025-04-08 20:25:51.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:51.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:52.061 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample' and table_type='BASE TABLE' 2025-04-08 20:25:52.078 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample 2025-04-08 20:25:52.079 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.079 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:52.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:52.091 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250401 2025-04-08 20:25:52.091 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.092 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:25:52.259 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.260 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:52.278 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:52.279 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250402 2025-04-08 20:25:52.279 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.279 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:25:52.734 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.735 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:52.753 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:52.754 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250403 2025-04-08 20:25:52.754 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.754 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:25:52.854 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:52.874 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:52.875 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250404 2025-04-08 20:25:52.875 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:52.876 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:25:53.037 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:53.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:53.054 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:53.055 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250405 2025-04-08 20:25:53.055 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:53.055 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:25:53.057 INFO 3009027 --- [ocp-schedules-11,041a530e18443a4a,d8eacd3d4d473f91] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:25:53.057 INFO 3009027 --- [ocp-schedules-19,3e9552248bc37f0d,fb516ac4e1ffcc2e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:25:53.059 INFO 3009027 --- [ocp-schedules-13,00101a1a1390f426,758b531ff46bf8ce] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:25:53.059 INFO 3009027 --- [ocp-schedules-12,1e986d7bf0912db7,dc88b3ccda15f8d7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:25:53.068 INFO 3009027 --- [ocp-schedules-23,689d16fa89c31bf1,31612b690bdc2552] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:25:53.075 INFO 3009027 --- [ocp-schedules-25,457b643d47a9f3ac,8b66ddf49fc87e2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:25:53.076 INFO 3009027 --- [ocp-async-7,8d17c89f54adf548,1c79752423d657ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:25:53.081 WARN 3009027 --- [ocp-async-2,0c843521cab6ec10,0d21195eefaf1e25] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:53.081 INFO 3009027 --- [ocp-async-2,905c3e2cd74c5a9d,9145263beaa691ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:53.104 INFO 3009027 --- [ocp-schedules-20,327ff9110935c611,4546c6e4981ed41a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:25:53.165 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:53.166 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:25:53.177746+08:00, lastProcessTime=2025-04-08T20:25:43.178616+08:00 2025-04-08 20:25:53.331 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:25:53.177746+08:00 2025-04-08 20:25:53.351 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:53.352 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:53.370 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:53.371 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250406 2025-04-08 20:25:53.371 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:53.371 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:25:53.520 INFO 3009027 --- [ocp-schedules-15,9cd6194df5237883,30ebcc20a4e256e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:25:53.681 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,f26d1c6c90c52935,942411686ef7f17b] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=f26d1c6c90c52935, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744115153620797,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 20:25:53.700 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,f26d1c6c90c52935,942411686ef7f17b] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=f26d1c6c90c52935, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744115153620797,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=78 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 20:25:53.708 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:53.709 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:53.727 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:53.727 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250407 2025-04-08 20:25:53.728 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:53.728 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:25:53.857 INFO 3009027 --- [ocp-schedules-21,0a2d41b95fdc446f,c284a9a8dff3cdd7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:25:53.858 INFO 3009027 --- [ocp-schedules-21,0a2d41b95fdc446f,c284a9a8dff3cdd7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:25:54.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:54.099 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:54.099 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250408 2025-04-08 20:25:54.100 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.100 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:25:54.235 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.236 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:54.254 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:54.255 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250409 2025-04-08 20:25:54.255 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.256 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:25:54.391 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.392 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:54.408 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:54.409 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250410 2025-04-08 20:25:54.409 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.410 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:25:54.569 INFO 3009027 --- [prometheus-minute-schedule-1,67ddda4d42c418b6,14e33acbda1d36ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:54.683 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.684 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:54.701 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:54.701 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250411 2025-04-08 20:25:54.702 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:54.702 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:25:55.002 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.003 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:55.023 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:55.023 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250412 2025-04-08 20:25:55.024 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.024 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:25:55.081 WARN 3009027 --- [ocp-async-2,c3a24989e80f0bd6,7c50515dbe4cd62f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:55.081 INFO 3009027 --- [ocp-async-3,4217e71a4cd1a389,f1155122120bdb56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:55.220 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.221 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:55.238 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:55.239 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250413 2025-04-08 20:25:55.239 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:25:55.561 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.562 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:55.579 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:55.580 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250414 2025-04-08 20:25:55.580 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.581 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:25:55.803 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.804 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:55.828 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:55.828 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250415 2025-04-08 20:25:55.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:25:55.949 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.950 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:55.968 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:55.969 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics, partition=P20250416 2025-04-08 20:25:55.969 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:55.969 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_audit_sample_statistics` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:25:56.166 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:56.166 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.167 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:56.167 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_audit_sample_statistics' and table_type='BASE TABLE' 2025-04-08 20:25:56.185 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_audit_sample_statistics 2025-04-08 20:25:56.186 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.186 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:25:56.186 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.187 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 20:25:56.190 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250309 2025-04-08 20:25:56.191 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.191 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250309 values less than (1741564800000000)) 2025-04-08 20:25:56.537 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.538 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:56.554 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:56.555 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250310 2025-04-08 20:25:56.555 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.556 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250310 values less than (1741651200000000)) 2025-04-08 20:25:56.817 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.818 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:56.836 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:56.837 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250311 2025-04-08 20:25:56.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:56.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250311 values less than (1741737600000000)) 2025-04-08 20:25:57.081 WARN 3009027 --- [ocp-async-1,ccf835dc7d99a374,b5028c69a047d2bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:57.081 INFO 3009027 --- [ocp-async-5,978a8ea341db1332,665fe80e6e464df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:57.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.163 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:57.179 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:57.180 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250312 2025-04-08 20:25:57.180 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.180 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250312 values less than (1741824000000000)) 2025-04-08 20:25:57.363 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.364 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:57.383 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:57.384 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250313 2025-04-08 20:25:57.385 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.385 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250313 values less than (1741910400000000)) 2025-04-08 20:25:57.569 INFO 3009027 --- [prometheus-minute-schedule-1,1242d4ac6f6df26d,f0c9361d9073dd63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115160, interval=60 2025-04-08 20:25:57.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:57.595 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:57.596 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250314 2025-04-08 20:25:57.596 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.597 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250314 values less than (1741996800000000)) 2025-04-08 20:25:57.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:57.845 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:57.846 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250315 2025-04-08 20:25:57.846 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.847 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250315 values less than (1742083200000000)) 2025-04-08 20:25:57.985 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:57.986 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:58.003 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:58.004 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250316 2025-04-08 20:25:58.005 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.005 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250316 values less than (1742169600000000)) 2025-04-08 20:25:58.323 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.324 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:58.340 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:58.341 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250317 2025-04-08 20:25:58.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250317 values less than (1742256000000000)) 2025-04-08 20:25:58.481 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.482 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:58.498 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:58.498 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250318 2025-04-08 20:25:58.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250318 values less than (1742342400000000)) 2025-04-08 20:25:58.710 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.711 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:58.727 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:58.728 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250319 2025-04-08 20:25:58.728 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.729 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250319 values less than (1742428800000000)) 2025-04-08 20:25:58.923 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.924 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:58.943 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:58.944 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250320 2025-04-08 20:25:58.944 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:58.945 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250320 values less than (1742515200000000)) 2025-04-08 20:25:59.082 WARN 3009027 --- [ocp-async-9,198e96dafec06c5b,813bb64c51c6898b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:25:59.082 INFO 3009027 --- [ocp-async-6,14d051eae1242cd0,033bfc19f5a5de4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:25:59.167 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:25:59.168 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:25:59.205 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.207 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:59.223 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:59.224 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250321 2025-04-08 20:25:59.224 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.224 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250321 values less than (1742601600000000)) 2025-04-08 20:25:59.365 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.366 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:59.382 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:59.382 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250322 2025-04-08 20:25:59.383 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.383 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250322 values less than (1742688000000000)) 2025-04-08 20:25:59.493 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.494 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:59.510 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:59.511 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250323 2025-04-08 20:25:59.511 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.511 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250323 values less than (1742774400000000)) 2025-04-08 20:25:59.720 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.721 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:59.736 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:59.737 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250324 2025-04-08 20:25:59.737 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.738 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:25:59.867 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.868 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:59.885 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:59.885 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250325 2025-04-08 20:25:59.886 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.886 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:25:59.947 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:25:59.966 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:25:59.967 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250326 2025-04-08 20:25:59.967 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:25:59.968 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:26:00.272 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:00.273 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:00.291 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:00.292 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250327 2025-04-08 20:26:00.292 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:00.293 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:26:00.569 INFO 3009027 --- [prometheus-minute-schedule-1,d2673ef55886933f,7d360882ca68642f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:00.808 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:00.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:00.827 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:00.827 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250328 2025-04-08 20:26:00.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:00.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:26:00.956 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:00.958 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:00.976 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:00.976 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250329 2025-04-08 20:26:00.977 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:00.977 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:26:01.082 WARN 3009027 --- [ocp-async-2,767d553b881b4da2,24504058cb47e23f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:01.082 INFO 3009027 --- [ocp-async-10,1f40a2843731642e,23bb4f960f34987f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:01.208 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:01.209 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:01.227 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:01.227 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250330 2025-04-08 20:26:01.228 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:01.228 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:26:01.507 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:01.508 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:01.526 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:01.527 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250331 2025-04-08 20:26:01.527 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:01.528 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:26:01.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:01.856 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:01.871 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:01.872 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250401 2025-04-08 20:26:01.872 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:01.873 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:26:02.004 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.005 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:02.024 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:02.025 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250402 2025-04-08 20:26:02.025 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.026 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:26:02.168 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:02.168 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:02.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.224 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:02.240 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:02.241 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250403 2025-04-08 20:26:02.241 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.242 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:26:02.609 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.610 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:02.626 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:02.627 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250404 2025-04-08 20:26:02.627 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.628 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:26:02.775 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:02.794 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:02.794 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250405 2025-04-08 20:26:02.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:02.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:26:03.018 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.019 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:03.037 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:03.038 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250406 2025-04-08 20:26:03.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.039 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:26:03.076 INFO 3009027 --- [ocp-schedules-10,e103dd3dd75613af,6c63db38abdc2976] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:26:03.076 INFO 3009027 --- [ocp-async-8,50b550cad79e640d,75e3c72220d66a8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:26:03.082 WARN 3009027 --- [ocp-async-1,08e09ab58ba9332e,5e0f2475cc3df270] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:03.082 INFO 3009027 --- [ocp-async-3,26ce904ed3d32a00,1b59f49c3dc3fe82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:03.140 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.141 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:03.160 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:03.161 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250407 2025-04-08 20:26:03.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.162 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:26:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:26:03.176872+08:00, lastProcessTime=2025-04-08T20:25:53.177746+08:00 2025-04-08 20:26:03.264 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.265 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:03.281 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:03.282 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250408 2025-04-08 20:26:03.282 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.282 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:26:03.299 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:26:03.176872+08:00 2025-04-08 20:26:03.395 INFO 3009027 --- [ocp-schedules-28,90840bc8175c0977,0bec8d4b74208223] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:26:03.397 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.398 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:03.414 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:03.415 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250409 2025-04-08 20:26:03.416 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.416 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:26:03.477 INFO 3009027 --- [ocp-schedules-28,90840bc8175c0977,0bec8d4b74208223] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:26:03.477 INFO 3009027 --- [ocp-schedules-28,90840bc8175c0977,0bec8d4b74208223] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:26:03.543 INFO 3009027 --- [ocp-schedules-20,85e828a20a10d9de,335f2b572ae64cc2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:26:03.571 INFO 3009027 --- [prometheus-minute-schedule-1,43ba7a421eb14433,694e332ffe23e03c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:03.617 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.618 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:03.636 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:03.636 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250410 2025-04-08 20:26:03.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:26:03.913 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.914 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:03.931 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:03.931 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250411 2025-04-08 20:26:03.932 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:03.932 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:26:03.938 INFO 3009027 --- [ocp-schedules-17,62384c150f177be4,a2b2ca0289f387a5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:26:03.939 INFO 3009027 --- [ocp-schedules-17,62384c150f177be4,a2b2ca0289f387a5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:26:04.149 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.150 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:04.168 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:04.169 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250412 2025-04-08 20:26:04.169 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.170 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:26:04.482 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.483 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:04.498 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:04.499 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250413 2025-04-08 20:26:04.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.500 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:26:04.772 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.774 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:04.794 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:04.795 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250414 2025-04-08 20:26:04.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.796 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:26:04.949 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.950 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:04.966 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:04.967 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250415 2025-04-08 20:26:04.967 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:04.968 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:26:05.082 WARN 3009027 --- [ocp-async-5,96ed77103ba27f12,0d52ff90b77c6ffc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:05.082 INFO 3009027 --- [ocp-async-9,6787a17749ba8eea,004f733f6e881af8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:05.169 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:05.169 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:05.307 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:05.308 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:05.324 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:05.325 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis_task, partition=P20250416 2025-04-08 20:26:05.325 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:05.325 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis_task` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:26:05.755 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:05.756 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis_task' and table_type='BASE TABLE' 2025-04-08 20:26:05.772 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis_task 2025-04-08 20:26:05.773 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:05.773 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:05.774 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:05.774 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 20:26:05.778 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250309 2025-04-08 20:26:05.779 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:05.779 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250309 values less than (1741564800000000)) 2025-04-08 20:26:06.286 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:06.287 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:06.302 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:06.303 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250310 2025-04-08 20:26:06.303 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:06.303 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250310 values less than (1741651200000000)) 2025-04-08 20:26:06.569 INFO 3009027 --- [prometheus-minute-schedule-1,e7ab2dc39880a41d,9d7aed84050f51fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:06.630 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:06.631 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:06.647 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:06.648 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250311 2025-04-08 20:26:06.648 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:06.648 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250311 values less than (1741737600000000)) 2025-04-08 20:26:06.862 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:06.863 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:06.880 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:06.880 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250312 2025-04-08 20:26:06.881 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:06.881 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250312 values less than (1741824000000000)) 2025-04-08 20:26:07.050 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.051 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:07.067 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:07.068 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250313 2025-04-08 20:26:07.068 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.069 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250313 values less than (1741910400000000)) 2025-04-08 20:26:07.082 WARN 3009027 --- [ocp-async-10,ed69f6cefca4c00b,e1b4e677fc8cf2a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:07.082 INFO 3009027 --- [ocp-async-6,34e18ed5db5ec1c3,375aeef84cadd75b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:07.336 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:07.354 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:07.354 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250314 2025-04-08 20:26:07.355 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.355 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250314 values less than (1741996800000000)) 2025-04-08 20:26:07.739 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:07.757 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:07.758 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250315 2025-04-08 20:26:07.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250315 values less than (1742083200000000)) 2025-04-08 20:26:07.944 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.944 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:07.961 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:07.961 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250316 2025-04-08 20:26:07.962 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:07.962 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250316 values less than (1742169600000000)) 2025-04-08 20:26:08.169 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:08.170 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:08.219 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:08.220 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:08.236 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:08.237 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250317 2025-04-08 20:26:08.237 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:08.238 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250317 values less than (1742256000000000)) 2025-04-08 20:26:08.600 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:08.601 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:08.617 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:08.618 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250318 2025-04-08 20:26:08.618 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:08.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250318 values less than (1742342400000000)) 2025-04-08 20:26:08.876 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:08.877 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:08.893 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:08.894 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250319 2025-04-08 20:26:08.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:08.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250319 values less than (1742428800000000)) 2025-04-08 20:26:09.082 WARN 3009027 --- [ocp-async-4,421db95999699417,3c200fc1af56fafb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:09.082 INFO 3009027 --- [ocp-async-8,67936d56adca0a6d,f618f0c2d84e2f78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:09.140 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:09.141 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:09.157 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:09.158 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250320 2025-04-08 20:26:09.158 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:09.159 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250320 values less than (1742515200000000)) 2025-04-08 20:26:09.569 INFO 3009027 --- [prometheus-minute-schedule-1,93668b912f22afae,00486b47d38badfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:09.618 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:09.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:09.634 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:09.635 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250321 2025-04-08 20:26:09.636 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:09.636 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250321 values less than (1742601600000000)) 2025-04-08 20:26:09.992 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:09.993 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:10.001 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:26:10.001 INFO 3009027 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:26:10.009 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:10.009 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250322 2025-04-08 20:26:10.010 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:10.010 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250322 values less than (1742688000000000)) 2025-04-08 20:26:10.138 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 0, ruleSize = 0. 2025-04-08 20:26:10.198 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:10.199 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:10.216 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:10.217 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250323 2025-04-08 20:26:10.217 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:10.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250323 values less than (1742774400000000)) 2025-04-08 20:26:10.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:10.696 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:10.714 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:10.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250324 2025-04-08 20:26:10.715 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:10.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:26:10.983 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:10.984 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:11.000 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:11.001 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250325 2025-04-08 20:26:11.001 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.001 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:26:11.082 WARN 3009027 --- [ocp-async-8,b366caf6ea98f5b5,fa50347691bcb9fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:11.082 INFO 3009027 --- [ocp-async-5,f261b4d518df7157,628f2afeb0722507] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:11.170 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:11.171 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:11.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.390 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:11.407 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:11.408 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250326 2025-04-08 20:26:11.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:26:11.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.656 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:11.674 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:11.674 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250327 2025-04-08 20:26:11.675 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.675 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:26:11.820 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.821 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:11.836 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:11.837 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250328 2025-04-08 20:26:11.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:11.838 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:26:12.239 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:12.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:12.256 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:12.257 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250329 2025-04-08 20:26:12.257 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:12.258 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:26:12.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:12.535 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:12.552 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:12.552 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250330 2025-04-08 20:26:12.553 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:12.553 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:26:12.569 INFO 3009027 --- [prometheus-minute-schedule-1,b092271719546795,0e4129c6247feab7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:12.570 INFO 3009027 --- [prometheus-cache-monitor1,538d9c4dd340a773,5e51218887cd37d2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:26:12.570 INFO 3009027 --- [prometheus-cache-monitor1,538d9c4dd340a773,5e51218887cd37d2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 20:26:13.076 INFO 3009027 --- [ocp-schedules-8,2faf97be81cd8680,16c47820d942d449] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:26:13.076 INFO 3009027 --- [ocp-async-5,dd0645921f009683,6e6b4b695396a692] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:26:13.083 WARN 3009027 --- [ocp-async-7,4cd6d9214108e7a5,d72eb51d227039d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:13.083 INFO 3009027 --- [ocp-async-9,d587dd28f025869b,5ebf641f56548096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:13.143 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:13.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:13.162 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:13.163 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250331 2025-04-08 20:26:13.164 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:13.164 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:26:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:26:13.176669+08:00, lastProcessTime=2025-04-08T20:26:03.176872+08:00 2025-04-08 20:26:13.402 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:26:13.176669+08:00 2025-04-08 20:26:13.571 INFO 3009027 --- [ocp-schedules-14,28bcb31214199d9e,67e6541732cee306] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:26:13.697 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:13.698 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:13.714 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:13.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250401 2025-04-08 20:26:13.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:13.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:26:14.052 INFO 3009027 --- [ocp-schedules-29,137b9bad9c29c6bc,df80ca06c18cf314] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:26:14.053 INFO 3009027 --- [ocp-schedules-29,137b9bad9c29c6bc,df80ca06c18cf314] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:26:14.171 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:14.172 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:14.265 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:14.266 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:14.282 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:14.282 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250402 2025-04-08 20:26:14.283 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:14.283 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:26:14.449 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:14.450 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:14.468 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:14.468 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250403 2025-04-08 20:26:14.469 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:14.469 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:26:14.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:14.760 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:14.780 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:14.781 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250404 2025-04-08 20:26:14.781 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:14.782 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:26:15.083 WARN 3009027 --- [ocp-async-1,363ce378866574a1,ef24753012be0303] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:15.083 INFO 3009027 --- [ocp-async-8,652bdb3545128ff5,8ef9153e900091e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:15.418 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:15.419 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:15.436 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:15.437 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250405 2025-04-08 20:26:15.437 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:15.438 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:26:15.569 INFO 3009027 --- [prometheus-minute-schedule-1,4d299fe08843d1d3,86411f62c1460259] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:15.654 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:15.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:15.671 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:15.672 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250406 2025-04-08 20:26:15.672 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:15.673 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:26:16.089 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:16.090 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:16.107 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:16.108 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250407 2025-04-08 20:26:16.108 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:16.109 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:26:16.669 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:16.670 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:16.687 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:16.688 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250408 2025-04-08 20:26:16.688 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:16.688 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:26:17.083 WARN 3009027 --- [ocp-async-9,a29be19b208ec5cd,bb0a1e02d7d96ede] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:17.083 INFO 3009027 --- [ocp-async-3,a353d2da45832772,fa1df7b654c9e355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:17.172 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:17.173 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:17.174 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:17.175 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:17.191 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:17.192 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250409 2025-04-08 20:26:17.192 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:17.192 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:26:17.570 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:17.571 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:17.586 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:17.587 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250410 2025-04-08 20:26:17.587 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:17.587 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:26:17.756 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:17.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:17.773 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:17.774 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250411 2025-04-08 20:26:17.774 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:17.775 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:26:18.288 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:18.289 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:18.305 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:18.305 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250412 2025-04-08 20:26:18.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:18.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:26:18.569 INFO 3009027 --- [prometheus-minute-schedule-1,5f4c0ac73e44f224,c22d4c9899a24227] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:18.633 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:18.634 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:18.650 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:18.651 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250413 2025-04-08 20:26:18.652 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:18.652 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:26:19.083 WARN 3009027 --- [ocp-async-3,d8b248f9a3fa5ca5,55d8814c05a0a645] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:19.083 INFO 3009027 --- [ocp-async-6,ad859b610803007c,8870ec38579e8f04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:19.089 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:19.089 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:19.106 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:19.107 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250414 2025-04-08 20:26:19.107 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:19.107 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:26:19.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:19.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:19.594 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:19.595 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250415 2025-04-08 20:26:19.595 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:19.596 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:26:20.097 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.098 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:20.114 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:20.115 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_perf_sql_diagnosis, partition=P20250416 2025-04-08 20:26:20.115 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.115 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_perf_sql_diagnosis` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:26:20.173 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:20.173 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:20.256 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.257 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_perf_sql_diagnosis' and table_type='BASE TABLE' 2025-04-08 20:26:20.273 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_perf_sql_diagnosis 2025-04-08 20:26:20.274 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.274 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:20.274 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.274 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 20:26:20.280 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250331 2025-04-08 20:26:20.280 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.280 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:26:20.444 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.445 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:20.462 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:20.462 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250401 2025-04-08 20:26:20.463 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:20.463 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:26:21.083 WARN 3009027 --- [ocp-async-6,93087fefbc367e56,2293f574a6f4e014] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:21.083 INFO 3009027 --- [ocp-async-5,84bcba0db8dba278,a0a6add3d5e98276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:21.100 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.101 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:21.118 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:21.119 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250402 2025-04-08 20:26:21.119 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.120 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:26:21.352 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.353 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:21.369 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:21.369 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250403 2025-04-08 20:26:21.370 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.370 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:26:21.569 INFO 3009027 --- [prometheus-minute-schedule-1,fe705809fb44bdcc,4c792ac445bac14f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:21.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:21.594 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:21.595 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250404 2025-04-08 20:26:21.595 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.596 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:26:21.780 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.781 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:21.797 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:21.798 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250405 2025-04-08 20:26:21.798 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.799 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:26:21.931 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.932 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:21.949 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:21.949 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250406 2025-04-08 20:26:21.950 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:21.950 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:26:22.294 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.295 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:22.312 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:22.312 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250407 2025-04-08 20:26:22.313 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.313 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:26:22.406 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.407 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:22.423 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:22.424 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250408 2025-04-08 20:26:22.424 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.425 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:26:22.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.638 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:22.654 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:22.654 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250409 2025-04-08 20:26:22.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:26:22.824 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.825 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:22.842 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:22.842 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250410 2025-04-08 20:26:22.843 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:22.843 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:26:23.035 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.036 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:23.046 INFO 3009027 --- [ocp-schedules-29,41b25549a36e0532,fe9107215c69329d] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T20:26:23.046+08:00, tenantSize=0 2025-04-08 20:26:23.054 INFO 3009027 --- [ocp-schedules-9,fb00a596e86894f6,b30013f15369b547] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:26:23.054+08:00, clusters=[] 2025-04-08 20:26:23.055 INFO 3009027 --- [ocp-schedules-9,fad8ba3684525c33,2e376be5b25948e3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:26:23.055+08:00, clusters=[] 2025-04-08 20:26:23.055 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:23.056 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250411 2025-04-08 20:26:23.056 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.056 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:26:23.057 INFO 3009027 --- [ocp-schedules-5,ba79fcd2467d6d04,ff1681b8be382845] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:26:23.057 INFO 3009027 --- [ocp-schedules-30,d25f72d05bcfda25,9ae0e63062a80a26] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:26:23.057 INFO 3009027 --- [ocp-schedules-8,3ae205c7c56bb956,f389fe9bfe438dbc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:26:23.059 INFO 3009027 --- [ocp-schedules-2,d2f7b880a84164ce,0db5ec86a15b3304] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:26:23.059 INFO 3009027 --- [ocp-schedules-3,b5e9b8839f9cee32,f903e239c3d324e7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:26:23.059 INFO 3009027 --- [ocp-schedules-26,d8cc2cb4875b9afd,ac1813541dfab284] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:26:23.060 INFO 3009027 --- [ocp-schedules-14,707bfd4921634af9,d2d10c0d93e89acb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:26:23.065 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:26:23.065 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:26:23.068 INFO 3009027 --- [ocp-schedules-25,e4d5708e59382e8a,9349f94f66b0d3b1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 20:26:23.069 INFO 3009027 --- [ocp-schedules-25,e4d5708e59382e8a,9349f94f66b0d3b1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 20:26:23.071 INFO 3009027 --- [ocp-schedules-5,ba79fcd2467d6d04,ff1681b8be382845] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:26:23.071 INFO 3009027 --- [ocp-schedules-28,c5d5f4297571639c,f1365d40ee6343de] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:26:23.071 INFO 3009027 --- [ocp-schedules-2,d2f7b880a84164ce,0db5ec86a15b3304] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:26:23.072 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:26:23.072 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:26:23.072 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:26:23.074 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:26:23.074 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:26:23.074 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:26:23.074 INFO 3009027 --- [ocp-schedules-1,864203522f27d5a8,9191701461355cf8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":318,"seriesIdKeySize":318,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:26:23.074 INFO 3009027 --- [ocp-schedules-1,864203522f27d5a8,9191701461355cf8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=11, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, http_server_requests_seconds_count=11, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:26:23.076 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:26:23.076 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:26:23.076 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:26:23.076 INFO 3009027 --- [ocp-schedules-22,8eb3b884869d2c56,91f9690d0246c844] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:26:23.076 INFO 3009027 --- [ocp-async-8,010af203b35a78be,7834259174e52ce1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:26:23.079 INFO 3009027 --- [ocp-schedules-4,3d9b9515d78161ff,1768ddd166bfa7b6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:26:23.079 INFO 3009027 --- [ocp-schedules-12,09c23df2bbd4e9aa,713146b15a7b74ec] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:26:23.081 INFO 3009027 --- [ocp-schedules-15,84cbf921f65d8d50,c196d7145f3c5498] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:26:23.082 INFO 3009027 --- [ocp-schedules-20,5601ab2bb80f907f,5f24c26e3f348448] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:26:23.083 WARN 3009027 --- [ocp-async-7,5f6789f3de510922,a21f93be6c05d095] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:23.084 INFO 3009027 --- [ocp-async-3,7b04833a286b35a7,3c348b982324edc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:23.084 INFO 3009027 --- [ocp-schedules-20,5601ab2bb80f907f,5f24c26e3f348448] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:26:23.084 INFO 3009027 --- [ocp-schedules-13,01821dcd3b16fe5f,087c2028d44495e9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:26:23.085 INFO 3009027 --- [ocp-schedules-13,01821dcd3b16fe5f,087c2028d44495e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:26:23.085 INFO 3009027 --- [ocp-schedules-17,cec6b42503b798d5,a70d3fddd9194b7f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:26:23.085 INFO 3009027 --- [ocp-schedules-17,cec6b42503b798d5,a70d3fddd9194b7f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:26:23.088 INFO 3009027 --- [ocp-schedules-8,996f72927ec94df9,9aeb4239610eb7b3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:26:23.089 INFO 3009027 --- [ocp-schedules-3,c82d60dd2dab3f72,cc29c6ff678e1cbb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:26:23.091 INFO 3009027 --- [ocp-schedules-19,355bc82e3827f60c,b82f8a6e0e30a7b9] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 20:26:23.091 INFO 3009027 --- [ocp-schedules-19,355bc82e3827f60c,b82f8a6e0e30a7b9] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 20:26:23.103 INFO 3009027 --- [ocp-schedules-14,f0c7647f864b470d,853b42d3f89bc0af] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:26:23.106 INFO 3009027 --- [ocp-schedules-30,2edc13814ed1e0cf,63553edf43055b28] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:26:23.117 INFO 3009027 --- [ocp-schedules-18,b3260b683c66cb9d,7519cfa289882675] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:26:23.117 INFO 3009027 --- [ocp-schedules-18,b3260b683c66cb9d,7519cfa289882675] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:26:23.118 INFO 3009027 --- [ocp-schedules-18,b3260b683c66cb9d,7519cfa289882675] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:26:23.118 INFO 3009027 --- [ocp-schedules-18,b3260b683c66cb9d,7519cfa289882675] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:26:23.121 INFO 3009027 --- [ocp-schedules-2,86a11acdc2887f08,0309ed6d8e019c4b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:26:23.121 INFO 3009027 --- [ocp-schedules-2,86a11acdc2887f08,0309ed6d8e019c4b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:26:23.144 INFO 3009027 --- [ocp-schedules-28,ee4e3bbc702d015b,51265642242fa51b] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:26:23.144 INFO 3009027 --- [ocp-schedules-28,ee4e3bbc702d015b,51265642242fa51b] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:26:23.145 INFO 3009027 --- [ocp-schedules-7,afd8ebb2cb35f699,51d799d105604e5b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:26:23.146 INFO 3009027 --- [ocp-schedules-7,afd8ebb2cb35f699,51d799d105604e5b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:26:23.146 INFO 3009027 --- [ocp-schedules-7,afd8ebb2cb35f699,51d799d105604e5b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:26:23.174 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:23.174 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:26:23.176348+08:00, lastProcessTime=2025-04-08T20:26:13.176669+08:00 2025-04-08 20:26:23.197 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:26:23.197 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:26:23.197 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:26:23.197 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:26:23.211 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=318, maxSyncedRecordId=766197260254310400 2025-04-08 20:26:23.211 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:26:23.211 INFO 3009027 --- [ocp-schedules-22,64d23082f2161fff,c427fa3c4180d06f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:26:23.322 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:26:23.176348+08:00 2025-04-08 20:26:23.329 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.330 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:23.347 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:23.348 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250412 2025-04-08 20:26:23.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:26:23.470 INFO 3009027 --- [ocp-schedules-15,8c60a46cb261036c,0bb96797cc0a9e72] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:26:23.478 INFO 3009027 --- [ocp-schedules-20,d1d4fc13e37e3c1d,f499e437a45d24f2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:26:23.559 INFO 3009027 --- [ocp-schedules-20,d1d4fc13e37e3c1d,f499e437a45d24f2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:26:23.559 INFO 3009027 --- [ocp-schedules-20,d1d4fc13e37e3c1d,f499e437a45d24f2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:26:23.596 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.598 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:23.601 INFO 3009027 --- [ocp-schedules-17,b5e93dff65f9bad4,139971571aaa5522] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:26:23.618 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:23.619 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250413 2025-04-08 20:26:23.620 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.621 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:26:23.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.838 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:23.854 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:23.855 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250414 2025-04-08 20:26:23.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.856 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:26:23.906 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.907 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:23.924 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:23.925 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250415 2025-04-08 20:26:23.925 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:23.925 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:26:24.126 INFO 3009027 --- [ocp-schedules-2,dcaee362b4871bd7,2bdebbc3d75a1794] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:26:24.127 INFO 3009027 --- [ocp-schedules-2,dcaee362b4871bd7,2bdebbc3d75a1794] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:26:24.187 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:24.204 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:24.205 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_active_session_stat_0, partition=P20250416 2025-04-08 20:26:24.206 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.206 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_active_session_stat_0` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:26:24.494 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.495 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_active_session_stat_0' and table_type='BASE TABLE' 2025-04-08 20:26:24.510 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_active_session_stat_0 2025-04-08 20:26:24.511 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.512 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:24.512 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.512 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 20:26:24.517 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202501 2025-04-08 20:26:24.518 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.518 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:24.569 INFO 3009027 --- [prometheus-minute-schedule-1,410657dda59339aa,f0cf55f4ff742ac0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:24.697 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.698 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2025-04-08 20:26:24.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2025-04-08 20:26:24.716 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202502 2025-04-08 20:26:24.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:24.804 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.805 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2025-04-08 20:26:24.821 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2025-04-08 20:26:24.822 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202503 2025-04-08 20:26:24.822 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.823 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:24.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.977 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2025-04-08 20:26:24.993 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2025-04-08 20:26:24.994 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_snapshot, partition=P202504 2025-04-08 20:26:24.995 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:24.995 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_snapshot` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:25.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.082 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_snapshot' and table_type='BASE TABLE' 2025-04-08 20:26:25.084 WARN 3009027 --- [ocp-async-2,7c8562f7336a4f16,e762026f553eb542] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:25.084 INFO 3009027 --- [ocp-async-3,0db7c385181321f0,4f9b866747abf82a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:25.098 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_snapshot 2025-04-08 20:26:25.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:25.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.100 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 20:26:25.103 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202501 2025-04-08 20:26:25.103 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.103 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:25.355 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.356 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2025-04-08 20:26:25.372 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2025-04-08 20:26:25.373 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202502 2025-04-08 20:26:25.373 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.373 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:25.483 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.484 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2025-04-08 20:26:25.499 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2025-04-08 20:26:25.500 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202503 2025-04-08 20:26:25.500 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.501 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:25.798 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.799 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2025-04-08 20:26:25.814 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2025-04-08 20:26:25.815 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cluster, partition=P202504 2025-04-08 20:26:25.815 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.816 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cluster` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:25.903 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cluster' and table_type='BASE TABLE' 2025-04-08 20:26:25.920 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cluster 2025-04-08 20:26:25.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:25.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 20:26:25.927 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202501 2025-04-08 20:26:25.927 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:25.927 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:26.136 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.137 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2025-04-08 20:26:26.153 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2025-04-08 20:26:26.154 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202502 2025-04-08 20:26:26.154 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.154 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:26.174 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:26.175 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:26.343 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.343 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2025-04-08 20:26:26.360 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2025-04-08 20:26:26.361 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202503 2025-04-08 20:26:26.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:26.505 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.506 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2025-04-08 20:26:26.523 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2025-04-08 20:26:26.524 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_tenant, partition=P202504 2025-04-08 20:26:26.524 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.524 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_tenant` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:26.862 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.863 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_tenant' and table_type='BASE TABLE' 2025-04-08 20:26:26.879 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_tenant 2025-04-08 20:26:26.880 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.880 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:26.880 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.881 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 20:26:26.885 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202501 2025-04-08 20:26:26.885 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:26.886 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:27.084 WARN 3009027 --- [ocp-async-3,39e5fe8ea6558f23,78ad35cf50d3cade] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:27.084 INFO 3009027 --- [ocp-async-5,c8936ebc3f2556c5,9d44424cc8c70358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:27.139 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:26:27.317 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:27.318 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2025-04-08 20:26:27.334 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2025-04-08 20:26:27.335 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202502 2025-04-08 20:26:27.335 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:27.336 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:27.569 INFO 3009027 --- [prometheus-minute-schedule-1,a15612db3cbdb20c,db2031c8e0797d13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:27.598 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:27.599 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2025-04-08 20:26:27.615 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2025-04-08 20:26:27.616 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202503 2025-04-08 20:26:27.616 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:27.616 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:28.077 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:28.078 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2025-04-08 20:26:28.096 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2025-04-08 20:26:28.096 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_server, partition=P202504 2025-04-08 20:26:28.096 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:28.097 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_server` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:28.739 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:28.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_server' and table_type='BASE TABLE' 2025-04-08 20:26:28.756 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_server 2025-04-08 20:26:28.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:28.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:28.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:28.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 20:26:28.760 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202501 2025-04-08 20:26:28.761 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:28.761 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:29.084 WARN 3009027 --- [ocp-async-4,24eb6440e1f79adb,9436f46692d90b24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:29.084 INFO 3009027 --- [ocp-async-10,ef48b1bfa8372c58,f17e4732e5169586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:29.175 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:29.176 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:29.354 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:29.355 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2025-04-08 20:26:29.372 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2025-04-08 20:26:29.373 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202502 2025-04-08 20:26:29.373 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:29.374 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:29.833 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:29.834 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2025-04-08 20:26:29.849 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2025-04-08 20:26:29.850 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202503 2025-04-08 20:26:29.850 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:29.851 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:30.288 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:30.289 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2025-04-08 20:26:30.305 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2025-04-08 20:26:30.305 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_replica, partition=P202504 2025-04-08 20:26:30.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:30.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_replica` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:30.569 INFO 3009027 --- [prometheus-minute-schedule-1,460ab19eddf2d150,c3ebda9e21e5b09e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:30.946 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:30.947 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_replica' and table_type='BASE TABLE' 2025-04-08 20:26:30.962 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_replica 2025-04-08 20:26:30.963 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:30.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:30.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:30.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 20:26:30.968 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202501 2025-04-08 20:26:30.968 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:30.968 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:31.084 WARN 3009027 --- [ocp-async-1,fd54c98d01232b82,78b4d67ec965c1c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:31.084 INFO 3009027 --- [ocp-async-10,bd499deda92ae947,253c2854eef2feca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:31.282 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:31.283 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2025-04-08 20:26:31.301 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2025-04-08 20:26:31.302 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202502 2025-04-08 20:26:31.302 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:31.302 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:31.771 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:31.771 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2025-04-08 20:26:31.788 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2025-04-08 20:26:31.789 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202503 2025-04-08 20:26:31.789 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:31.789 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:32.176 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:32.177 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:32.491 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:32.492 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2025-04-08 20:26:32.509 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2025-04-08 20:26:32.510 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sysstat, partition=P202504 2025-04-08 20:26:32.510 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:32.510 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sysstat` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:32.989 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:32.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sysstat' and table_type='BASE TABLE' 2025-04-08 20:26:33.006 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sysstat 2025-04-08 20:26:33.007 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.007 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:33.008 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.008 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 20:26:33.011 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202501 2025-04-08 20:26:33.011 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.011 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:33.064 WARN 3009027 --- [ocp-schedules-24,ded78f183fde0873,944c84e497297c47] c.o.o.c.d.lock.jdbc.DefaultLockManager : LockException while executing with jdbc lock part-worker Cannot acquire lock for key part-worker 2025-04-08 20:26:33.064 WARN 3009027 --- [ocp-schedules-24,ded78f183fde0873,944c84e497297c47] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=partitionDaemon, confName=partitionDaemon 2025-04-08 20:26:33.065 ERROR 3009027 --- [ocp-schedules-24,ded78f183fde0873,944c84e497297c47] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.distributed.lock.LockException: Cannot acquire lock for key part-worker at com.oceanbase.ocp.core.distributed.lock.jdbc.DefaultLockManager.doExecuteWhileLocked(DefaultLockManager.java:97) at com.oceanbase.ocp.core.distributed.lock.jdbc.DefaultLockManager.executeWhileLocked(DefaultLockManager.java:81) at com.oceanbase.ocp.service.partitioning.PartitionRolloverService.watchAllWhileLocked(PartitionRolloverService.java:121) at com.oceanbase.ocp.server.common.scheduled.OcpSchedules.partitionDaemon(OcpSchedules.java:130) ... 15 common frames omitted 2025-04-08 20:26:33.076 INFO 3009027 --- [ocp-schedules-7,e3e6536a6daa1885,ca6c336a71256eb7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:26:33.076 INFO 3009027 --- [ocp-async-3,8cdf20cf82ec2d17,71d2ac864274b467] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:26:33.084 WARN 3009027 --- [ocp-async-9,1fac6763bef468d0,313c399dcb16e45f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:33.084 INFO 3009027 --- [ocp-async-4,a62e9c51902af5dc,2f1798b09c90536b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:26:33.176798+08:00, lastProcessTime=2025-04-08T20:26:23.176348+08:00 2025-04-08 20:26:33.372 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:26:33.176798+08:00 2025-04-08 20:26:33.415 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.415 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2025-04-08 20:26:33.432 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2025-04-08 20:26:33.433 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202502 2025-04-08 20:26:33.433 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.433 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:33.569 INFO 3009027 --- [prometheus-minute-schedule-1,c0f84ea4052b767c,0a6c84b947866487] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:33.654 INFO 3009027 --- [ocp-schedules-22,966af6b1ea2c2042,71c78fed6c9aff9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:26:33.978 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.979 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2025-04-08 20:26:33.996 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2025-04-08 20:26:33.996 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202503 2025-04-08 20:26:33.997 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:33.997 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:34.284 INFO 3009027 --- [ocp-schedules-24,f654b05bc70fab78,43f4b3c788d3ba51] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:26:34.286 INFO 3009027 --- [ocp-schedules-24,f654b05bc70fab78,43f4b3c788d3ba51] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:26:34.357 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:34.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2025-04-08 20:26:34.375 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2025-04-08 20:26:34.375 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_row_lock, partition=P202504 2025-04-08 20:26:34.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:34.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_row_lock` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:34.756 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:34.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_row_lock' and table_type='BASE TABLE' 2025-04-08 20:26:34.775 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_row_lock 2025-04-08 20:26:34.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:34.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:34.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:34.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 20:26:34.780 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202501 2025-04-08 20:26:34.781 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:34.781 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:35.084 WARN 3009027 --- [ocp-async-4,3c2583724cb762f6,69363a35f1ac62b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:35.085 INFO 3009027 --- [ocp-async-1,974c9bc7f5040980,cc1fe873f7ee3845] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:35.177 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:35.178 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:35.355 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:35.356 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2025-04-08 20:26:35.375 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2025-04-08 20:26:35.375 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202502 2025-04-08 20:26:35.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:35.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:35.845 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:35.846 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2025-04-08 20:26:35.862 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2025-04-08 20:26:35.863 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202503 2025-04-08 20:26:35.863 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:35.863 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:36.200 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:36.200 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2025-04-08 20:26:36.217 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2025-04-08 20:26:36.218 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_latch, partition=P202504 2025-04-08 20:26:36.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:36.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_latch` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:36.569 INFO 3009027 --- [prometheus-minute-schedule-1,ea081ca1a86011a1,cf6790a2c6e9c059] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:36.824 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:36.825 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_latch' and table_type='BASE TABLE' 2025-04-08 20:26:36.842 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_latch 2025-04-08 20:26:36.843 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:36.843 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:36.844 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:36.844 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 20:26:36.847 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202501 2025-04-08 20:26:36.848 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:36.848 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:37.085 WARN 3009027 --- [ocp-async-9,5b9dd0ae906490f8,c453649b393836a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:37.085 INFO 3009027 --- [ocp-async-8,85be3770a99cf52b,00288b66125a89b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:37.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:37.390 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2025-04-08 20:26:37.407 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2025-04-08 20:26:37.408 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202502 2025-04-08 20:26:37.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:37.409 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:37.876 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:37.877 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2025-04-08 20:26:37.893 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2025-04-08 20:26:37.894 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202503 2025-04-08 20:26:37.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:37.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:38.178 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:38.179 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:38.423 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:38.424 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2025-04-08 20:26:38.441 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2025-04-08 20:26:38.441 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_cache, partition=P202504 2025-04-08 20:26:38.442 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:38.442 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_cache` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:39.031 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.032 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_cache' and table_type='BASE TABLE' 2025-04-08 20:26:39.048 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_cache 2025-04-08 20:26:39.049 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.049 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:39.049 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.049 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 20:26:39.053 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202501 2025-04-08 20:26:39.053 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.053 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:39.085 WARN 3009027 --- [ocp-async-5,45ceee498592d6ba,9f0559135ce72a58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:39.085 INFO 3009027 --- [ocp-async-7,06c5c87fa0393cb6,5ebe920b94595bd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:39.410 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.411 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2025-04-08 20:26:39.428 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2025-04-08 20:26:39.428 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202502 2025-04-08 20:26:39.429 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.429 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:39.569 INFO 3009027 --- [prometheus-minute-schedule-1,12cbb26e43009d97,47a407ea66faf624] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:39.831 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.832 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2025-04-08 20:26:39.848 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2025-04-08 20:26:39.848 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202503 2025-04-08 20:26:39.849 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:39.849 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:40.001 INFO 3009027 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:26:40.002 INFO 3009027 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:26:40.395 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:40.396 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2025-04-08 20:26:40.414 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2025-04-08 20:26:40.415 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_parameter, partition=P202504 2025-04-08 20:26:40.415 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:40.415 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_parameter` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:40.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:40.759 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_parameter' and table_type='BASE TABLE' 2025-04-08 20:26:40.775 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_parameter 2025-04-08 20:26:40.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:40.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:40.776 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:40.777 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 20:26:40.780 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202501 2025-04-08 20:26:40.780 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:40.781 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:41.085 WARN 3009027 --- [ocp-async-7,475a0c21056367cd,36685ce40b711738] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:41.085 INFO 3009027 --- [ocp-async-2,d8e06c858ae997f4,e7cde6e180b7fd97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:41.114 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:26:41.116 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=2ms, period=120000ms 2025-04-08 20:26:41.179 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:41.180 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:41.294 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:41.295 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2025-04-08 20:26:41.311 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2025-04-08 20:26:41.311 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202502 2025-04-08 20:26:41.312 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:41.312 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:41.818 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:41.819 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2025-04-08 20:26:41.834 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2025-04-08 20:26:41.835 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202503 2025-04-08 20:26:41.835 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:41.836 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:42.076 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.076 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2025-04-08 20:26:42.093 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2025-04-08 20:26:42.094 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_rpc, partition=P202504 2025-04-08 20:26:42.094 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.094 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_rpc` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:42.408 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.409 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_rpc' and table_type='BASE TABLE' 2025-04-08 20:26:42.424 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_rpc 2025-04-08 20:26:42.425 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:42.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 20:26:42.430 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202501 2025-04-08 20:26:42.430 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.430 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:42.569 INFO 3009027 --- [prometheus-minute-schedule-1,b77162d13ef4a356,5a6cb9b4a7999d7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:42.989 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:42.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2025-04-08 20:26:43.008 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2025-04-08 20:26:43.009 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202502 2025-04-08 20:26:43.009 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:43.009 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:43.076 INFO 3009027 --- [ocp-schedules-29,60bf942fa48972c1,70304940637384cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:26:43.077 INFO 3009027 --- [ocp-async-3,d9d61f4b622b09ea,548b093df9b5c3d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:26:43.085 WARN 3009027 --- [ocp-async-9,c2a6d8af1ade5a0f,cd5b1f299358393b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:43.085 INFO 3009027 --- [ocp-async-1,6acf2a5c1015fa28,44539b9d20ae8a36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:26:43.177558+08:00, lastProcessTime=2025-04-08T20:26:33.176798+08:00 2025-04-08 20:26:43.408 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:26:43.177558+08:00 2025-04-08 20:26:43.560 INFO 3009027 --- [ocp-schedules-18,c8bb468d091ada01,09416e3df60b0c7e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:26:43.676 INFO 3009027 --- [ocp-schedules-22,6dac168dcd467f3f,1db64e151d7ba2f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:26:43.696 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:43.697 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2025-04-08 20:26:43.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2025-04-08 20:26:43.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202503 2025-04-08 20:26:43.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:43.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:43.722 INFO 3009027 --- [ocp-schedules-18,c8bb468d091ada01,09416e3df60b0c7e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:26:43.722 INFO 3009027 --- [ocp-schedules-18,c8bb468d091ada01,09416e3df60b0c7e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:26:44.097 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:44.098 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2025-04-08 20:26:44.114 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2025-04-08 20:26:44.114 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_packet, partition=P202504 2025-04-08 20:26:44.115 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:44.115 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_packet` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:44.180 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:44.181 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:44.470 INFO 3009027 --- [ocp-schedules-29,97faa32c3cf01420,9d20506c9446cad3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:26:44.472 INFO 3009027 --- [ocp-schedules-29,97faa32c3cf01420,9d20506c9446cad3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:26:44.836 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:44.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_packet' and table_type='BASE TABLE' 2025-04-08 20:26:44.854 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_packet 2025-04-08 20:26:44.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:44.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:44.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:44.856 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 20:26:44.859 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202501 2025-04-08 20:26:44.859 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:44.860 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:45.085 WARN 3009027 --- [ocp-async-7,f2c5a45d868c2809,9103cce1b78d1c67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:45.085 INFO 3009027 --- [ocp-async-8,045e97b6ef84458b,aa938acc643bc5d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:45.451 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:45.452 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2025-04-08 20:26:45.468 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2025-04-08 20:26:45.469 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202502 2025-04-08 20:26:45.469 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:45.469 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:45.569 INFO 3009027 --- [prometheus-minute-schedule-1,b07e14b39277209f,aefecf9791548450] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:45.971 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:45.972 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2025-04-08 20:26:45.989 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2025-04-08 20:26:45.989 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202503 2025-04-08 20:26:45.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:45.990 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:46.472 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:46.473 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2025-04-08 20:26:46.493 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2025-04-08 20:26:46.494 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_clog, partition=P202504 2025-04-08 20:26:46.495 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:46.495 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_clog` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:46.896 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:46.897 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_clog' and table_type='BASE TABLE' 2025-04-08 20:26:46.915 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_clog 2025-04-08 20:26:46.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:46.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:46.917 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:46.917 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 20:26:46.921 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202501 2025-04-08 20:26:46.921 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:46.921 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:47.086 WARN 3009027 --- [ocp-async-1,b6ec3830b90decdc,6ebf2679cd1421fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:47.086 INFO 3009027 --- [ocp-async-9,c0443624949d9c23,5892087c91c428d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:47.181 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:47.182 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:47.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:47.342 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2025-04-08 20:26:47.359 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2025-04-08 20:26:47.360 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202502 2025-04-08 20:26:47.360 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:47.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:47.765 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:47.766 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2025-04-08 20:26:47.782 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2025-04-08 20:26:47.783 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202503 2025-04-08 20:26:47.783 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:47.784 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:48.375 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:48.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2025-04-08 20:26:48.394 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2025-04-08 20:26:48.394 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_location_cache, partition=P202504 2025-04-08 20:26:48.395 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:48.395 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_location_cache` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:48.569 INFO 3009027 --- [prometheus-minute-schedule-1,8b175045eaa25104,4fe5fd5cd31b1f34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:48.919 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:48.919 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_location_cache' and table_type='BASE TABLE' 2025-04-08 20:26:48.935 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_location_cache 2025-04-08 20:26:48.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:48.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:48.937 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:48.937 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 20:26:48.941 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202501 2025-04-08 20:26:48.941 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:48.942 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:49.086 WARN 3009027 --- [ocp-async-5,5ec2efd0199ab75a,da752db74c0e0983] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:49.086 INFO 3009027 --- [ocp-async-9,4252443934799651,c8785ad59fbad413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:49.600 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:49.600 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2025-04-08 20:26:49.617 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2025-04-08 20:26:49.618 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202502 2025-04-08 20:26:49.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:49.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:50.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.100 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2025-04-08 20:26:50.118 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2025-04-08 20:26:50.119 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202503 2025-04-08 20:26:50.119 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.119 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:50.182 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:50.183 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:50.513 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.514 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2025-04-08 20:26:50.530 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2025-04-08 20:26:50.531 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_gts, partition=P202504 2025-04-08 20:26:50.531 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.531 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_gts` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:50.765 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.766 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_gts' and table_type='BASE TABLE' 2025-04-08 20:26:50.783 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_gts 2025-04-08 20:26:50.784 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.784 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:50.784 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.784 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 20:26:50.788 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202501 2025-04-08 20:26:50.788 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:50.788 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:51.086 WARN 3009027 --- [ocp-async-7,117d74fd4f67915c,8eebdefa7714fdff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:51.086 INFO 3009027 --- [ocp-async-9,32aee00f576da836,06c66bc15487f111] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:51.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:51.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2025-04-08 20:26:51.239 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2025-04-08 20:26:51.240 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202502 2025-04-08 20:26:51.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:51.241 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:51.569 INFO 3009027 --- [prometheus-minute-schedule-1,d18a0dc182d1c01a,8e884a4663e04fce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:51.642 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:51.643 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2025-04-08 20:26:51.659 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2025-04-08 20:26:51.660 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202503 2025-04-08 20:26:51.660 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:51.660 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:51.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:51.965 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2025-04-08 20:26:51.981 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2025-04-08 20:26:51.982 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_system_event, partition=P202504 2025-04-08 20:26:51.982 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:51.982 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_system_event` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:52.362 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:52.363 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_system_event' and table_type='BASE TABLE' 2025-04-08 20:26:52.379 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_system_event 2025-04-08 20:26:52.380 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:52.380 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:52.380 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:52.380 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 20:26:52.384 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202501 2025-04-08 20:26:52.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:52.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:53.057 INFO 3009027 --- [ocp-schedules-21,d4d635905efc7cd2,e0e86f017bc4a98c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:26:53.057 INFO 3009027 --- [ocp-schedules-11,2e87701640d72f9d,eac2edb74bf645e0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:26:53.058 INFO 3009027 --- [ocp-schedules-16,f74faf2aead20874,cbb65a20ff673915] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:26:53.058 INFO 3009027 --- [ocp-schedules-13,7b5d649a8a0c7b57,5d965ac4f0cb6b7d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:26:53.060 INFO 3009027 --- [ocp-schedules-8,b4fbcd7eabd72054,fce2460edb6474a9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:26:53.060 INFO 3009027 --- [ocp-schedules-8,b4fbcd7eabd72054,fce2460edb6474a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:26:53.060 INFO 3009027 --- [ocp-schedules-27,f4c6a6f751ec9fd2,e204995eee83a71f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:26:53.060 INFO 3009027 --- [ocp-schedules-27,f4c6a6f751ec9fd2,e204995eee83a71f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:26:53.060 INFO 3009027 --- [ocp-schedules-28,fb80e01924286721,c8b3858d7dfd4f58] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:26:53.060 INFO 3009027 --- [ocp-schedules-28,fb80e01924286721,c8b3858d7dfd4f58] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:26:53.064 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:53.065 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2025-04-08 20:26:53.078 INFO 3009027 --- [ocp-schedules-18,d0e8aeee0d15fec5,8b3b5313c7d3fd5b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:26:53.078 INFO 3009027 --- [ocp-async-1,51fc0c5f838880bc,932dae268b9d59d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:26:53.081 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2025-04-08 20:26:53.081 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202502 2025-04-08 20:26:53.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:53.082 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:53.086 WARN 3009027 --- [ocp-async-4,7e6a1bd54c654218,28483dc39dbbecd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:53.086 INFO 3009027 --- [ocp-async-9,0fcd10f7313fc9e5,a9e9454365279cd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:53.106 INFO 3009027 --- [ocp-schedules-6,e3a4ff92eb42b5ba,ea95508c79347bb8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:26:53.107 INFO 3009027 --- [ocp-schedules-29,364fa21aa38e2d19,8979ffe82d3b7218] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:26:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:26:53.178300+08:00, lastProcessTime=2025-04-08T20:26:43.177558+08:00 2025-04-08 20:26:53.183 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:53.183 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:53.340 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:26:53.178300+08:00 2025-04-08 20:26:53.455 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:53.455 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2025-04-08 20:26:53.471 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2025-04-08 20:26:53.472 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202503 2025-04-08 20:26:53.472 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:53.473 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:53.759 INFO 3009027 --- [ocp-schedules-23,a772db86bfe2e16a,b1164b28bb6d70bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:26:53.893 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:53.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2025-04-08 20:26:53.911 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2025-04-08 20:26:53.911 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_memory, partition=P202504 2025-04-08 20:26:53.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:53.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_memory` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:54.439 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:54.440 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_memory' and table_type='BASE TABLE' 2025-04-08 20:26:54.456 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_memory 2025-04-08 20:26:54.457 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:54.457 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:54.457 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:54.457 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 20:26:54.460 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202501 2025-04-08 20:26:54.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:54.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202501 values less than (1738368000000000)) 2025-04-08 20:26:54.544 INFO 3009027 --- [ocp-schedules-4,5644a60b413ebbc2,1636ecc837b269ce] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:26:54.545 INFO 3009027 --- [ocp-schedules-4,5644a60b413ebbc2,1636ecc837b269ce] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:26:54.569 INFO 3009027 --- [prometheus-minute-schedule-1,c5d88e03971084b9,edc2853e88713b20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:54.917 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:54.918 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2025-04-08 20:26:54.934 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2025-04-08 20:26:54.935 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202502 2025-04-08 20:26:54.935 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:54.936 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202502 values less than (1740787200000000)) 2025-04-08 20:26:55.086 WARN 3009027 --- [ocp-async-9,93e773d421b1b616,39de4a54ab66ee89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:55.086 INFO 3009027 --- [ocp-async-10,3319140a5245b5f5,e739d24346b5adac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:55.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:55.307 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2025-04-08 20:26:55.324 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2025-04-08 20:26:55.324 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202503 2025-04-08 20:26:55.324 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:55.325 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202503 values less than (1743465600000000)) 2025-04-08 20:26:55.810 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:55.811 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2025-04-08 20:26:55.827 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2025-04-08 20:26:55.828 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_hit, partition=P202504 2025-04-08 20:26:55.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:55.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_hit` add partition(partition P202504 values less than (1746057600000000)) 2025-04-08 20:26:56.183 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:56.184 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:56.206 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:56.207 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_hit' and table_type='BASE TABLE' 2025-04-08 20:26:56.224 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_hit 2025-04-08 20:26:56.225 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:56.225 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:26:56.225 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:56.225 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 20:26:56.232 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250324 2025-04-08 20:26:56.232 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:56.233 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250324 values less than (1742860800000000)) 2025-04-08 20:26:56.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:56.717 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:26:56.734 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:26:56.734 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250325 2025-04-08 20:26:56.735 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:56.735 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250325 values less than (1742947200000000)) 2025-04-08 20:26:57.086 WARN 3009027 --- [ocp-async-6,e6cccfb70654e222,e2d1e63c37ac783c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:57.086 INFO 3009027 --- [ocp-async-10,d5bdcf9a43753f31,4ef9034475859b0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:57.318 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:57.319 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:26:57.336 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:26:57.337 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250326 2025-04-08 20:26:57.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:57.338 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:26:57.569 INFO 3009027 --- [prometheus-minute-schedule-1,eac01f74d0a73362,20dd5268ef9bf6d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115220, interval=60 2025-04-08 20:26:58.362 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1025, sql=alter table `ob_hist_sql_plan_monitor` add partition(partition P20250326 values less than (1743033600000000)) 2025-04-08 20:26:58.362 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:58.363 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:26:58.382 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:26:58.383 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250327 2025-04-08 20:26:58.383 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:58.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250327 values less than (1743120000000000)) 2025-04-08 20:26:59.087 WARN 3009027 --- [ocp-async-8,ba17db50da75bd3e,12da51f23cdbd666] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:26:59.087 INFO 3009027 --- [ocp-async-10,90d288cedae6db89,be5e7713cf9401d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:26:59.185 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:26:59.186 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:26:59.305 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:59.306 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:26:59.322 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:26:59.323 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250328 2025-04-08 20:26:59.323 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:26:59.323 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250328 values less than (1743206400000000)) 2025-04-08 20:27:00.008 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:00.009 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:00.026 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:00.027 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250329 2025-04-08 20:27:00.028 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:00.028 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250329 values less than (1743292800000000)) 2025-04-08 20:27:00.569 INFO 3009027 --- [prometheus-minute-schedule-1,ea2ad318ff673adf,fd0de21328f9d883] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:00.731 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:00.733 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:00.753 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:00.753 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250330 2025-04-08 20:27:00.754 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:00.754 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250330 values less than (1743379200000000)) 2025-04-08 20:27:01.087 WARN 3009027 --- [ocp-async-9,f40b72d5723e4fd5,7ee83896e0fe58c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:01.087 INFO 3009027 --- [ocp-async-10,c1476d41d678f15e,78561d0bce8e80e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:01.112 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:01.113 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:01.129 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:01.130 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250331 2025-04-08 20:27:01.130 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:01.131 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250331 values less than (1743465600000000)) 2025-04-08 20:27:01.622 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:01.622 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:01.639 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:01.640 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250401 2025-04-08 20:27:01.640 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:01.641 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:27:02.186 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:02.187 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:02.321 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:02.322 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:02.339 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:02.339 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250402 2025-04-08 20:27:02.340 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:02.340 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:27:03.078 INFO 3009027 --- [ocp-schedules-19,c281f50de72b26c8,ea6809c7adab29fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:27:03.078 INFO 3009027 --- [ocp-async-2,9f14912fda7eda55,472ce6aa117bf7b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:27:03.087 WARN 3009027 --- [ocp-async-10,f7b0f09221d142bd,1e69247bfcf68b72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:03.087 INFO 3009027 --- [ocp-async-3,ff7f207a115f3c0b,20e87a334c55f12d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:27:03.177975+08:00, lastProcessTime=2025-04-08T20:26:53.178300+08:00 2025-04-08 20:27:03.222 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:03.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:03.239 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:03.239 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250403 2025-04-08 20:27:03.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:03.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:27:03.355 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:27:03.177975+08:00 2025-04-08 20:27:03.569 INFO 3009027 --- [prometheus-minute-schedule-1,aed3920097ebd7c1,1046fb4b24810c0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:03.723 INFO 3009027 --- [ocp-schedules-5,72410607fdccfc50,a9c30ad57ccfbebf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:27:03.778 INFO 3009027 --- [ocp-schedules-13,3f0fef1500d27428,b9fb3bd6814bc402] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:27:03.904 INFO 3009027 --- [ocp-schedules-5,72410607fdccfc50,a9c30ad57ccfbebf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:27:03.904 INFO 3009027 --- [ocp-schedules-5,72410607fdccfc50,a9c30ad57ccfbebf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:27:03.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:03.931 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:03.947 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:03.948 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250404 2025-04-08 20:27:03.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:03.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:27:04.432 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:04.433 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:04.449 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:04.450 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250405 2025-04-08 20:27:04.450 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:04.451 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:27:04.692 INFO 3009027 --- [ocp-schedules-6,db01af3a2fafb127,a5548a9c22f39980] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:27:04.693 INFO 3009027 --- [ocp-schedules-6,db01af3a2fafb127,a5548a9c22f39980] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:27:05.087 WARN 3009027 --- [ocp-async-2,272534dad956da5b,5d2c416b90a1b50a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:05.087 INFO 3009027 --- [ocp-async-1,cbd1a2eb5dd5c25e,c11af662f1c06518] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:05.173 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:05.174 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:05.187 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:05.187 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:05.190 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:05.190 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250406 2025-04-08 20:27:05.191 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:05.191 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:27:05.749 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:05.750 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:05.766 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:05.767 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250407 2025-04-08 20:27:05.767 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:05.768 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:27:06.532 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:06.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:06.550 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:06.550 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250408 2025-04-08 20:27:06.551 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:06.551 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:27:06.569 INFO 3009027 --- [prometheus-minute-schedule-1,ad0286535e7cc4b8,1783ae1084eb2e00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:07.087 WARN 3009027 --- [ocp-async-7,b7910063a6d0b7e2,a1f9a9240d31273b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:07.087 INFO 3009027 --- [ocp-async-10,b8e4117201caf732,f832874411b9e74c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:07.317 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:07.318 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:07.334 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:07.334 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250409 2025-04-08 20:27:07.335 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:07.335 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:27:07.926 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:07.927 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:07.945 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:07.946 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250410 2025-04-08 20:27:07.947 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:07.947 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:27:08.188 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:08.189 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:08.296 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:08.297 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:08.313 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:08.314 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250411 2025-04-08 20:27:08.315 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:08.316 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:27:08.937 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:08.938 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:08.954 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:08.955 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250412 2025-04-08 20:27:08.955 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:08.956 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:27:09.087 WARN 3009027 --- [ocp-async-6,10b2cf991c52bbef,bed3fd268d8a29b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:09.087 INFO 3009027 --- [ocp-async-8,7963d101d40a78e9,d0e47402d4f28aa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:09.569 INFO 3009027 --- [prometheus-minute-schedule-1,6373124fdda6cecd,70219c5a48a66e8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:09.826 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:09.827 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:09.843 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:09.844 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250413 2025-04-08 20:27:09.845 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:09.845 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:27:10.001 INFO 3009027 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:27:10.001 INFO 3009027 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:27:10.138 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 0, ruleSize = 0. 2025-04-08 20:27:10.530 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:10.531 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:10.547 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:10.547 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250414 2025-04-08 20:27:10.548 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:10.548 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:27:11.088 WARN 3009027 --- [ocp-async-2,469104b2727e0b32,55adb8f7b54aad2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:11.088 INFO 3009027 --- [ocp-async-5,40246a6348bf55b6,ce41acf2fa72e097] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:11.189 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:11.190 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:11.220 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:11.221 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:11.238 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:11.239 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_sql_plan_monitor, partition=P20250415 2025-04-08 20:27:11.239 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:11.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_sql_plan_monitor` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:27:11.950 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:11.951 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_sql_plan_monitor' and table_type='BASE TABLE' 2025-04-08 20:27:11.968 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_sql_plan_monitor 2025-04-08 20:27:11.969 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:11.969 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:11.969 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:11.969 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 20:27:11.975 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250401 2025-04-08 20:27:11.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:11.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250401 values less than (1743552000000000)) 2025-04-08 20:27:12.511 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:23:12.415, end=2025-04-08T20:25:12.415} 2025-04-08 20:27:12.519 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:27:12.521 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:27:12.545 ERROR 3009027 --- [metric-archive-second-9,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489556) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489556) Table has no partition for value 2025-04-08 20:27:12.546 ERROR 3009027 --- [metric-archive-second-14,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489551) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489551) Table has no partition for value 2025-04-08 20:27:12.546 ERROR 3009027 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489552) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489552) Table has no partition for value 2025-04-08 20:27:12.547 ERROR 3009027 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489553) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489553) Table has no partition for value 2025-04-08 20:27:12.547 ERROR 3009027 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489555) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489555) Table has no partition for value 2025-04-08 20:27:12.547 ERROR 3009027 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489554) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489554) Table has no partition for value 2025-04-08 20:27:12.548 ERROR 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489551) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489551) Table has no partition for value at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1578) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:669) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:693) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1036) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1090) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=3221489551) Table has no partition for value at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1050) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=3221489551) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-04-08 20:27:12.548 ERROR 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489556) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489556) Table has no partition for value at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1578) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:669) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:693) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1036) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1090) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=3221489556) Table has no partition for value at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1050) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=3221489556) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-04-08 20:27:12.548 ERROR 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489552) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489552) Table has no partition for value at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1578) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:669) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:693) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1036) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1090) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=3221489552) Table has no partition for value at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1050) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=3221489552) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-04-08 20:27:12.549 ERROR 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489555) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489555) Table has no partition for value at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1578) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:669) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:693) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1036) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1090) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=3221489555) Table has no partition for value at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1050) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=3221489555) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-04-08 20:27:12.549 ERROR 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489554) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489554) Table has no partition for value at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1578) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:669) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:693) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1036) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1090) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=3221489554) Table has no partition for value at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1050) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=3221489554) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-04-08 20:27:12.549 ERROR 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [1526]; (conn=3221489553) Table has no partition for value; nested exception is java.sql.BatchUpdateException: (conn=3221489553) Table has no partition for value at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1578) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:669) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:693) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1036) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1090) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=3221489553) Table has no partition for value at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1050) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=3221489553) Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Table has no partition for value at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Table has no partition for value at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-04-08 20:27:12.569 INFO 3009027 --- [prometheus-minute-schedule-1,ebabdeb19eead66d,6c93f20a02c9dd0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:12.570 INFO 3009027 --- [prometheus-cache-monitor1,ed290442d6e47fb6,f6c59f099c83ce1f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":318,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":318,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:27:12.570 INFO 3009027 --- [prometheus-cache-monitor1,ed290442d6e47fb6,f6c59f099c83ce1f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=11, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, http_server_requests_seconds_count=11, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:27:12.857 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:12.858 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:12.874 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:12.875 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250402 2025-04-08 20:27:12.875 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:12.876 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250402 values less than (1743638400000000)) 2025-04-08 20:27:13.078 INFO 3009027 --- [ocp-schedules-29,2dbf665fe5cfdde4,6ed2a08ec6360f3c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:27:13.078 INFO 3009027 --- [ocp-async-7,5763054a6d37a002,3e1f7a15ebdadfcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:27:13.088 WARN 3009027 --- [ocp-async-4,ea099191e9294b8f,a839372310b4e02d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:13.088 INFO 3009027 --- [ocp-async-1,db27575d0823f69e,2094bb4251ef5cda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:27:13.176455+08:00, lastProcessTime=2025-04-08T20:27:03.177975+08:00 2025-04-08 20:27:13.357 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:27:13.176455+08:00 2025-04-08 20:27:13.643 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:13.644 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:13.659 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:13.660 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250403 2025-04-08 20:27:13.660 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:13.661 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250403 values less than (1743724800000000)) 2025-04-08 20:27:13.830 INFO 3009027 --- [ocp-schedules-14,a72dcb1aa4004428,65f30631e6d5f4de] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:27:14.190 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:14.191 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:14.456 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:14.458 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:14.473 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:14.474 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250404 2025-04-08 20:27:14.474 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:14.475 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250404 values less than (1743811200000000)) 2025-04-08 20:27:14.733 INFO 3009027 --- [ocp-schedules-28,601ab572a5309348,fd41bfcb738223ac] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:27:14.734 INFO 3009027 --- [ocp-schedules-28,601ab572a5309348,fd41bfcb738223ac] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:27:15.088 WARN 3009027 --- [ocp-async-7,65d3c6b1196bedd0,86123401c3bbfd4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:15.088 INFO 3009027 --- [ocp-async-3,32ca3886b8192f01,8739a02e807fa255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:15.389 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:15.390 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:15.406 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:15.406 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250405 2025-04-08 20:27:15.407 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:15.407 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250405 values less than (1743897600000000)) 2025-04-08 20:27:15.569 INFO 3009027 --- [prometheus-minute-schedule-1,8af813a6c4801c27,c93e2d4c8f8df1e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:16.131 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:16.132 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:16.147 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:16.148 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250406 2025-04-08 20:27:16.148 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:16.149 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250406 values less than (1743984000000000)) 2025-04-08 20:27:16.790 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:16.791 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:16.806 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:16.807 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250407 2025-04-08 20:27:16.807 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:16.808 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250407 values less than (1744070400000000)) 2025-04-08 20:27:17.088 WARN 3009027 --- [ocp-async-9,b6f7bb950d1ce1fc,3fc0afdbfc92a888] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:17.088 INFO 3009027 --- [ocp-async-8,bef807ff3ecd0916,8558c44656240d6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:17.191 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:17.192 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:17.585 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:17.586 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:17.603 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:17.603 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250408 2025-04-08 20:27:17.603 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:17.604 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250408 values less than (1744156800000000)) 2025-04-08 20:27:17.828 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:17.829 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:17.846 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:17.847 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250409 2025-04-08 20:27:17.847 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:17.847 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250409 values less than (1744243200000000)) 2025-04-08 20:27:18.086 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.087 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:18.103 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:18.104 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250410 2025-04-08 20:27:18.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250410 values less than (1744329600000000)) 2025-04-08 20:27:18.290 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.291 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:18.307 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:18.307 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250411 2025-04-08 20:27:18.308 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.308 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250411 values less than (1744416000000000)) 2025-04-08 20:27:18.519 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.520 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:18.537 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:18.537 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250412 2025-04-08 20:27:18.538 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.538 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250412 values less than (1744502400000000)) 2025-04-08 20:27:18.569 INFO 3009027 --- [prometheus-minute-schedule-1,3a3b4da8e42c7fbc,601e3c1914f66e27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:18.817 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.818 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:18.836 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:18.836 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250413 2025-04-08 20:27:18.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:18.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250413 values less than (1744588800000000)) 2025-04-08 20:27:19.050 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:19.050 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:19.066 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:19.067 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250414 2025-04-08 20:27:19.068 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:19.068 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250414 values less than (1744675200000000)) 2025-04-08 20:27:19.088 WARN 3009027 --- [ocp-async-8,3980f47c8c7937b3,7cb9ec9427f71222] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:19.088 INFO 3009027 --- [ocp-async-10,efbe8c972f38df14,243cdcaa443c4d58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:19.323 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:19.324 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:19.340 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:19.341 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250415 2025-04-08 20:27:19.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:19.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250415 values less than (1744761600000000)) 2025-04-08 20:27:19.620 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:19.621 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:19.638 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:19.639 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_trans_stat_v2, partition=P20250416 2025-04-08 20:27:19.639 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:19.640 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_trans_stat_v2` add partition(partition P20250416 values less than (1744848000000000)) 2025-04-08 20:27:20.015 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.016 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_trans_stat_v2' and table_type='BASE TABLE' 2025-04-08 20:27:20.032 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_trans_stat_v2 2025-04-08 20:27:20.033 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.033 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:20.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 20:27:20.038 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741449600000000 2025-04-08 20:27:20.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.039 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741449600000000 values less than (1741449600000000)) 2025-04-08 20:27:20.178 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.179 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:20.192 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:20.193 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:20.196 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:20.197 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741536000000000 2025-04-08 20:27:20.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.198 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741536000000000 values less than (1741536000000000)) 2025-04-08 20:27:20.386 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.387 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:20.403 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:20.404 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741622400000000 2025-04-08 20:27:20.404 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.404 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741622400000000 values less than (1741622400000000)) 2025-04-08 20:27:20.718 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.719 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:20.735 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:20.736 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741708800000000 2025-04-08 20:27:20.736 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:20.737 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741708800000000 values less than (1741708800000000)) 2025-04-08 20:27:21.059 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.060 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:21.076 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:21.077 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741795200000000 2025-04-08 20:27:21.077 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.078 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741795200000000 values less than (1741795200000000)) 2025-04-08 20:27:21.088 WARN 3009027 --- [ocp-async-2,735ab9e3d26bafe9,3b13e92670e2b138] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:21.088 INFO 3009027 --- [ocp-async-5,c4f138f7d57b9bd2,343de4cdcb15fca0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:21.198 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.199 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:21.214 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:21.215 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741881600000000 2025-04-08 20:27:21.216 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.216 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741881600000000 values less than (1741881600000000)) 2025-04-08 20:27:21.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.362 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:21.378 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:21.378 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1741968000000000 2025-04-08 20:27:21.379 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.379 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1741968000000000 values less than (1741968000000000)) 2025-04-08 20:27:21.569 INFO 3009027 --- [prometheus-minute-schedule-1,8d78b6fc01e62eda,cb1aed031a0b98f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:21.627 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.628 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:21.644 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:21.645 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742054400000000 2025-04-08 20:27:21.645 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.645 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742054400000000 values less than (1742054400000000)) 2025-04-08 20:27:21.928 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.929 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:21.944 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:21.945 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742140800000000 2025-04-08 20:27:21.945 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:21.946 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742140800000000 values less than (1742140800000000)) 2025-04-08 20:27:22.049 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.050 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:22.065 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:22.066 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742227200000000 2025-04-08 20:27:22.066 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.067 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742227200000000 values less than (1742227200000000)) 2025-04-08 20:27:22.240 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.241 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:22.257 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:22.258 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742313600000000 2025-04-08 20:27:22.258 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.259 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742313600000000 values less than (1742313600000000)) 2025-04-08 20:27:22.661 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.662 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:22.678 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:22.679 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742400000000000 2025-04-08 20:27:22.679 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.679 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742400000000000 values less than (1742400000000000)) 2025-04-08 20:27:22.882 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.883 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:22.899 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:22.900 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742486400000000 2025-04-08 20:27:22.900 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:22.900 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742486400000000 values less than (1742486400000000)) 2025-04-08 20:27:23.055 INFO 3009027 --- [ocp-schedules-24,e2447061ce223065,a277c32855f97170] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:27:23.055+08:00, clusters=[] 2025-04-08 20:27:23.056 INFO 3009027 --- [ocp-schedules-30,9ef6a7c4567b3bac,4eddaa41aa3fe14a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:27:23.056+08:00, clusters=[] 2025-04-08 20:27:23.057 INFO 3009027 --- [ocp-schedules-3,579b6b280df71c9c,d606f5f5cefe984e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:27:23.057 INFO 3009027 --- [ocp-schedules-9,9d79b91f227617cb,74c640f13024e33c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:27:23.057 INFO 3009027 --- [ocp-schedules-14,063d503ce2407fa9,5c8028e2451ce782] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:27:23.057 INFO 3009027 --- [ocp-schedules-8,791a4da155c5ea6e,b9fa2e16fbd79708] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:27:23.058 INFO 3009027 --- [ocp-schedules-7,285f237ea3a3d42f,8fe14d10ef82898c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:27:23.058 INFO 3009027 --- [ocp-schedules-13,188cc3ceb85cc9c3,4846b2f54d9e4299] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:27:23.061 INFO 3009027 --- [ocp-schedules-18,6dfeba50c8bad078,d7c15b8ebe3aebd0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:27:23.061 INFO 3009027 --- [ocp-schedules-18,6dfeba50c8bad078,d7c15b8ebe3aebd0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:27:23.061 INFO 3009027 --- [ocp-schedules-15,c39786eb3270b7f9,20dcce744d2ed8d7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:27:23.061 INFO 3009027 --- [ocp-schedules-15,c39786eb3270b7f9,20dcce744d2ed8d7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:27:23.061 INFO 3009027 --- [ocp-schedules-6,8185f32a48fb1650,1657432a689307c4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:27:23.061 INFO 3009027 --- [ocp-schedules-6,8185f32a48fb1650,1657432a689307c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:27:23.065 INFO 3009027 --- [ocp-schedules-8,791a4da155c5ea6e,b9fa2e16fbd79708] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:27:23.071 INFO 3009027 --- [ocp-schedules-19,d8d0bdda60910d31,2482586951811174] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 20:27:23.071 INFO 3009027 --- [ocp-schedules-19,d8d0bdda60910d31,2482586951811174] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 20:27:23.078 INFO 3009027 --- [ocp-schedules-27,6ff7eeed170cbca6,8f1058f37c5edd09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:27:23.078 INFO 3009027 --- [ocp-async-4,4ce84243575b9513,77416c917430cab3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:27:23.080 INFO 3009027 --- [ocp-schedules-26,c70e25616088f1c0,cebc5ab48e06e440] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:27:23.081 INFO 3009027 --- [ocp-schedules-25,be4e31568d358f02,e1734162fbb5887d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:27:23.088 WARN 3009027 --- [ocp-async-6,e92bcc17a05c476b,900928274cee6bad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:23.089 INFO 3009027 --- [ocp-async-3,a35b0e2b20013e44,ee88b04ae7ea21e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:23.090 INFO 3009027 --- [ocp-schedules-28,e199a0ac408a5e60,49bc1419085fd7cf] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:27:23.105 INFO 3009027 --- [ocp-schedules-7,b2656d2e0b3cb39e,3042bacd942bf9bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:27:23.108 INFO 3009027 --- [ocp-schedules-18,95de119f662fc3f5,ab460672e7741f8d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:27:23.118 INFO 3009027 --- [ocp-schedules-15,99e8956dac909603,a6969cf7fbc32bd8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:27:23.118 INFO 3009027 --- [ocp-schedules-15,99e8956dac909603,a6969cf7fbc32bd8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:27:23.119 INFO 3009027 --- [ocp-schedules-15,99e8956dac909603,a6969cf7fbc32bd8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:27:23.119 INFO 3009027 --- [ocp-schedules-15,99e8956dac909603,a6969cf7fbc32bd8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:27:23.125 INFO 3009027 --- [ocp-schedules-9,2ca3fdf35bdb696e,41209225c79cb1e3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:27:23.125 INFO 3009027 --- [ocp-schedules-9,2ca3fdf35bdb696e,41209225c79cb1e3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:27:23.146 INFO 3009027 --- [ocp-schedules-8,0b24b9e2a71757ff,0cd8d8bd2ef37e99] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:27:23.146 INFO 3009027 --- [ocp-schedules-8,0b24b9e2a71757ff,0cd8d8bd2ef37e99] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:27:23.146 INFO 3009027 --- [ocp-schedules-19,0fbb75618940929f,4844e7a3b4bee60f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:27:23.148 INFO 3009027 --- [ocp-schedules-19,0fbb75618940929f,4844e7a3b4bee60f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:27:23.148 INFO 3009027 --- [ocp-schedules-19,0fbb75618940929f,4844e7a3b4bee60f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:27:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:27:23.176398+08:00, lastProcessTime=2025-04-08T20:27:13.176455+08:00 2025-04-08 20:27:23.190 INFO 3009027 --- [ocp-schedules-10,e8e38f656150bc0c,9852e15b76287b7d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:27:23.193 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:23.193 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:23.212 INFO 3009027 --- [ocp-schedules-5,0e083f44575fc48b,35c8a503ea7349da] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:27:23.212 INFO 3009027 --- [ocp-schedules-5,0e083f44575fc48b,35c8a503ea7349da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:27:23.212 INFO 3009027 --- [ocp-schedules-5,0e083f44575fc48b,35c8a503ea7349da] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:27:23.212 INFO 3009027 --- [ocp-schedules-5,0e083f44575fc48b,35c8a503ea7349da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766197260254310400 2025-04-08 20:27:23.213 INFO 3009027 --- [ocp-schedules-5,0e083f44575fc48b,35c8a503ea7349da] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:27:23.213 INFO 3009027 --- [ocp-schedules-5,0e083f44575fc48b,35c8a503ea7349da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:27:23.230 INFO 3009027 --- [ocp-schedules-11,f3f4bf76cd629910,376387a04f6250d3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:27:23.230 INFO 3009027 --- [ocp-schedules-3,579b6b280df71c9c,d606f5f5cefe984e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:27:23.319 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:27:23.176398+08:00 2025-04-08 20:27:23.339 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.340 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:23.356 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:23.357 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742572800000000 2025-04-08 20:27:23.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742572800000000 values less than (1742572800000000)) 2025-04-08 20:27:23.524 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.525 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:23.541 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:23.542 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742659200000000 2025-04-08 20:27:23.542 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.543 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742659200000000 values less than (1742659200000000)) 2025-04-08 20:27:23.586 INFO 3009027 --- [ocp-schedules-2,584842686d978e22,f5c7d33eba37cda2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:27:23.791 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.792 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:23.808 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:23.808 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742745600000000 2025-04-08 20:27:23.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.809 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742745600000000 values less than (1742745600000000)) 2025-04-08 20:27:23.850 INFO 3009027 --- [ocp-schedules-4,4f47136ca6a29baa,5d5347df9685fe47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:27:23.907 INFO 3009027 --- [ocp-schedules-13,ca610a7e65531194,d1b6425aa8b611d5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:27:23.963 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.964 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:23.980 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:23.981 INFO 3009027 --- [ocp-schedules-13,ca610a7e65531194,d1b6425aa8b611d5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:27:23.981 INFO 3009027 --- [ocp-schedules-13,ca610a7e65531194,d1b6425aa8b611d5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:27:23.981 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742832000000000 2025-04-08 20:27:23.981 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:23.982 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742832000000000 values less than (1742832000000000)) 2025-04-08 20:27:24.330 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:24.331 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:24.346 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:24.347 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1742918400000000 2025-04-08 20:27:24.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:24.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1742918400000000 values less than (1742918400000000)) 2025-04-08 20:27:24.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:24.534 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:24.551 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:24.551 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743004800000000 2025-04-08 20:27:24.552 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:24.552 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743004800000000 values less than (1743004800000000)) 2025-04-08 20:27:24.569 INFO 3009027 --- [prometheus-minute-schedule-1,4847a375bd8bc37c,cc027c4bff451007] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:24.594 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:27:24.594 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:27:24.597 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:27:24.597 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:27:24.597 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:27:24.597 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:27:24.602 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:27:24.602 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:27:24.602 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:27:24.602 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:27:24.625 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:27:24.630 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:27:24.630 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:27:24.630 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:27:24.630 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:27:24.630 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:27:24.636 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:27:24.913 INFO 3009027 --- [ocp-schedules-1,af89ea476c0eeb5b,d29b59d9d77d71e7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:27:24.913 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:24.915 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:24.915 INFO 3009027 --- [ocp-schedules-1,af89ea476c0eeb5b,d29b59d9d77d71e7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:27:24.931 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:24.932 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743091200000000 2025-04-08 20:27:24.932 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:24.933 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743091200000000 values less than (1743091200000000)) 2025-04-08 20:27:24.973 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:27:25.061 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:25.079 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:25.079 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743177600000000 2025-04-08 20:27:25.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743177600000000 values less than (1743177600000000)) 2025-04-08 20:27:25.089 WARN 3009027 --- [ocp-async-7,5981826de66d1605,615496a48e4facac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:25.089 INFO 3009027 --- [ocp-async-3,a708bed98487da0d,545cf3aa29321c04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:25.293 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.691589587S 2025-04-08 20:27:25.294 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:27:25.351 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:27:25.351 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:27:25.377 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:27:25.418 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:27:25.418 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:27:25.418 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:27:25.418 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:27:25.418 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:27:25.419 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:27:25.425 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:25.426 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:27:25.426 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743264000000000 2025-04-08 20:27:25.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.426 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743264000000000 values less than (1743264000000000)) 2025-04-08 20:27:25.525 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] o.s.boot.SpringApplication : Started application in 0.927 seconds (JVM running for 286.98) 2025-04-08 20:27:25.525 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.928038383S 2025-04-08 20:27:25.738 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.739 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:25.755 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:25.756 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743350400000000 2025-04-08 20:27:25.756 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.756 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743350400000000 values less than (1743350400000000)) 2025-04-08 20:27:25.810 INFO 3009027 --- [ocp-schedules-12,1fecadc3ef639b18,07e424084351030d] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:27:25.911 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.913 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:25.928 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:25.929 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743436800000000 2025-04-08 20:27:25.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:25.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743436800000000 values less than (1743436800000000)) 2025-04-08 20:27:26.068 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:26.069 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:26.085 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:26.086 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743523200000000 2025-04-08 20:27:26.086 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:26.087 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743523200000000 values less than (1743523200000000)) 2025-04-08 20:27:26.193 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:26.194 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:26.566 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:26.567 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:26.583 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:26.584 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743609600000000 2025-04-08 20:27:26.584 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:26.585 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743609600000000 values less than (1743609600000000)) 2025-04-08 20:27:27.086 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:27.087 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:27.089 WARN 3009027 --- [ocp-async-6,d96751c41d09a3fe,ac634e5a65dbca22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:27.089 INFO 3009027 --- [ocp-async-3,92752e239e6cb599,bd4c9e728bd20943] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:27.107 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:27.107 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743696000000000 2025-04-08 20:27:27.108 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:27.108 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743696000000000 values less than (1743696000000000)) 2025-04-08 20:27:27.139 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:27:27.569 INFO 3009027 --- [prometheus-minute-schedule-1,dfe1fbd9db1e6a12,481a7374276c8137] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:27.692 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:27.693 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:27.708 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:27.709 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743782400000000 2025-04-08 20:27:27.709 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:27.710 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743782400000000 values less than (1743782400000000)) 2025-04-08 20:27:28.323 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:28.324 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:28.340 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:28.340 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743868800000000 2025-04-08 20:27:28.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:28.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743868800000000 values less than (1743868800000000)) 2025-04-08 20:27:29.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:29.035 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:29.051 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:29.052 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1743955200000000 2025-04-08 20:27:29.052 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:29.052 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1743955200000000 values less than (1743955200000000)) 2025-04-08 20:27:29.089 WARN 3009027 --- [ocp-async-5,cc72ca52f6fae4c4,30d66ce153e8b4b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:29.089 INFO 3009027 --- [ocp-async-3,900bdc40210f85d0,a9f57b5ca915cd10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:29.194 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:29.195 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:29.695 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:29.696 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:29.712 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:29.713 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1744041600000000 2025-04-08 20:27:29.713 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:29.714 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1744041600000000 values less than (1744041600000000)) 2025-04-08 20:27:30.363 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:30.364 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:30.380 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:30.381 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1744128000000000 2025-04-08 20:27:30.381 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:30.381 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1744128000000000 values less than (1744128000000000)) 2025-04-08 20:27:30.569 INFO 3009027 --- [prometheus-minute-schedule-1,571d8ac6b1161bab,4791dea1b413a326] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:31.089 WARN 3009027 --- [ocp-async-1,cea5e47e98fac631,85413de38c0d32e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:31.089 INFO 3009027 --- [ocp-async-8,71e20f410de02e40,ba2d4a3486bb4462] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:31.195 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:31.196 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:31.213 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:31.214 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1744214400000000 2025-04-08 20:27:31.214 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:31.215 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1744214400000000 values less than (1744214400000000)) 2025-04-08 20:27:31.860 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:31.861 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:31.877 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:31.878 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_system_event, partition=p1744300800000000 2025-04-08 20:27:31.878 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:31.879 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_system_event` add partition(partition p1744300800000000 values less than (1744300800000000)) 2025-04-08 20:27:32.195 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:32.196 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:32.467 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:32.468 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_system_event' and table_type='BASE TABLE' 2025-04-08 20:27:32.485 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_system_event 2025-04-08 20:27:32.486 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:32.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:32.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:32.487 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 20:27:32.491 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741449600000000 2025-04-08 20:27:32.492 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:32.492 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741449600000000 values less than (1741449600000000)) 2025-04-08 20:27:33.078 INFO 3009027 --- [ocp-schedules-30,6f40377d0c57b657,040d38f487d65219] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:27:33.079 INFO 3009027 --- [ocp-async-6,88b2f173cae6a131,469044c48fc2e846] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:27:33.089 WARN 3009027 --- [ocp-async-8,bfdd8ec28ac05939,8a77719baa444a18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:33.089 INFO 3009027 --- [ocp-async-10,ef8c53157527df79,b06a52540f07c973] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:33.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:27:33.177646+08:00, lastProcessTime=2025-04-08T20:27:23.176398+08:00 2025-04-08 20:27:33.365 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:27:33.177646+08:00 2025-04-08 20:27:33.429 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:33.430 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:33.446 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:33.446 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741536000000000 2025-04-08 20:27:33.447 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:33.447 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741536000000000 values less than (1741536000000000)) 2025-04-08 20:27:33.569 INFO 3009027 --- [prometheus-minute-schedule-1,9b70c95e688f59c9,992f7dba7aa0db17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:33.899 INFO 3009027 --- [ocp-schedules-26,e1ffc50ab8eeeb91,493935fbcaf1cfac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:27:33.940 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:33.941 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:33.956 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:33.957 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741622400000000 2025-04-08 20:27:33.957 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:33.958 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741622400000000 values less than (1741622400000000)) 2025-04-08 20:27:34.876 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:34.877 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:34.893 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:34.894 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741708800000000 2025-04-08 20:27:34.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:34.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741708800000000 values less than (1741708800000000)) 2025-04-08 20:27:34.967 INFO 3009027 --- [ocp-schedules-29,1adff92af276019e,23dd378798c1db04] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:27:34.969 INFO 3009027 --- [ocp-schedules-29,1adff92af276019e,23dd378798c1db04] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:27:35.089 WARN 3009027 --- [ocp-async-6,86c04d885448a596,6a1eae8e0502f444] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:35.090 INFO 3009027 --- [ocp-async-1,11045b720a1f103d,e5843bafe5b94f17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:35.196 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:35.196 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:35.932 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1036, sql=alter table `ob_cluster_server_event` add partition(partition p1741708800000000 values less than (1741708800000000)) 2025-04-08 20:27:35.932 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:35.933 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:35.948 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:35.949 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741795200000000 2025-04-08 20:27:35.949 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:35.949 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741795200000000 values less than (1741795200000000)) 2025-04-08 20:27:36.272 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:36.273 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:36.289 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:36.290 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741881600000000 2025-04-08 20:27:36.290 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:36.291 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741881600000000 values less than (1741881600000000)) 2025-04-08 20:27:36.491 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:36.492 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:36.508 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:36.509 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1741968000000000 2025-04-08 20:27:36.509 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:36.510 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1741968000000000 values less than (1741968000000000)) 2025-04-08 20:27:36.569 INFO 3009027 --- [prometheus-minute-schedule-1,6e6c1a2e63950345,f74aaf2e630bafc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:36.813 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:36.814 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:36.830 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:36.831 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742054400000000 2025-04-08 20:27:36.831 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:36.831 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742054400000000 values less than (1742054400000000)) 2025-04-08 20:27:37.090 WARN 3009027 --- [ocp-async-4,4c5e91cd9a3a4437,8d01932b5e0649c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:37.090 INFO 3009027 --- [ocp-async-8,f35e904aef298127,1df2da0c720832a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:37.174 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:37.175 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:37.195 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:37.196 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742140800000000 2025-04-08 20:27:37.196 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:37.197 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742140800000000 values less than (1742140800000000)) 2025-04-08 20:27:37.450 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:37.451 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:37.467 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:37.468 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742227200000000 2025-04-08 20:27:37.468 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:37.469 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742227200000000 values less than (1742227200000000)) 2025-04-08 20:27:37.701 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:37.701 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:37.718 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:37.718 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742313600000000 2025-04-08 20:27:37.719 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:37.719 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742313600000000 values less than (1742313600000000)) 2025-04-08 20:27:38.148 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:38.149 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:38.165 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:38.166 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742400000000000 2025-04-08 20:27:38.166 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:38.167 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742400000000000 values less than (1742400000000000)) 2025-04-08 20:27:38.197 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:38.197 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:38.714 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:38.715 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:38.731 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:38.732 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742486400000000 2025-04-08 20:27:38.732 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:38.733 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742486400000000 values less than (1742486400000000)) 2025-04-08 20:27:39.044 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:39.045 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:39.061 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:39.062 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742572800000000 2025-04-08 20:27:39.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:39.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742572800000000 values less than (1742572800000000)) 2025-04-08 20:27:39.090 WARN 3009027 --- [ocp-async-9,26f4604c1ebed1f8,fdfc2e7569a1cd4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:39.090 INFO 3009027 --- [ocp-async-1,f205c9bf8031a411,9362d03d7423b1c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:39.503 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:39.504 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:39.522 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:39.522 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742659200000000 2025-04-08 20:27:39.523 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:39.523 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742659200000000 values less than (1742659200000000)) 2025-04-08 20:27:39.569 INFO 3009027 --- [prometheus-minute-schedule-1,6739514a4cc30156,5fc4970235b85577] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:39.850 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:39.851 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:39.866 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:39.867 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742745600000000 2025-04-08 20:27:39.867 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:39.868 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742745600000000 values less than (1742745600000000)) 2025-04-08 20:27:40.002 INFO 3009027 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:27:40.002 INFO 3009027 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:27:40.114 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.115 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:40.132 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:40.133 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742832000000000 2025-04-08 20:27:40.133 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.133 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742832000000000 values less than (1742832000000000)) 2025-04-08 20:27:40.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.349 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:40.365 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:40.365 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1742918400000000 2025-04-08 20:27:40.366 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.366 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1742918400000000 values less than (1742918400000000)) 2025-04-08 20:27:40.645 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.646 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:40.663 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:40.663 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743004800000000 2025-04-08 20:27:40.664 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.664 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743004800000000 values less than (1743004800000000)) 2025-04-08 20:27:40.757 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.758 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:40.774 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:40.774 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743091200000000 2025-04-08 20:27:40.775 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:40.775 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743091200000000 values less than (1743091200000000)) 2025-04-08 20:27:41.009 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.010 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:41.025 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:41.025 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743177600000000 2025-04-08 20:27:41.025 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.026 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743177600000000 values less than (1743177600000000)) 2025-04-08 20:27:41.090 WARN 3009027 --- [ocp-async-10,88bbea9b4224dc9d,d77cceef7a0c33cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:41.090 INFO 3009027 --- [ocp-async-6,b54780fa04d67ba0,901a90306df498f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:41.197 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:41.198 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:41.217 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,b1f53ff510386ae9,d5e7475ccc3258f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /pre_login_loading?null, client=10.11.12.102, traceId=b1f53ff510386ae9, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.WebController.preLoginLoading(), args=] 2025-04-08 20:27:41.222 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,b1f53ff510386ae9,d5e7475ccc3258f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /pre_login_loading client=10.11.12.102, traceId=b1f53ff510386ae9, duration=12 ms] 2025-04-08 20:27:41.230 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.231 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:41.247 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:41.248 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743264000000000 2025-04-08 20:27:41.248 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.249 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743264000000000 values less than (1743264000000000)) 2025-04-08 20:27:41.367 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.368 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:41.385 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:41.386 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743350400000000 2025-04-08 20:27:41.386 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.387 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743350400000000 values less than (1743350400000000)) 2025-04-08 20:27:41.615 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.616 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:41.632 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:41.633 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743436800000000 2025-04-08 20:27:41.633 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:41.634 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743436800000000 values less than (1743436800000000)) 2025-04-08 20:27:42.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:42.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:42.178 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:42.178 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743523200000000 2025-04-08 20:27:42.179 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:42.179 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743523200000000 values less than (1743523200000000)) 2025-04-08 20:27:42.569 INFO 3009027 --- [prometheus-minute-schedule-1,f30bea5f0689b94c,c6b3809471c40aac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:42.830 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,e0f92fe886bea789,2a8acac411e92bcc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.11.12.102, traceId=e0f92fe886bea789, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-04-08 20:27:42.830 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,2a4242e54bf82f75,5d19a1c06d18d118] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.11.12.102, traceId=2a4242e54bf82f75, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2025-04-08 20:27:42.830 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,2a4242e54bf82f75,5d19a1c06d18d118] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.11.12.102, traceId=2a4242e54bf82f75, duration=3 ms] 2025-04-08 20:27:42.840 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,e0f92fe886bea789,2a8acac411e92bcc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.11.12.102, traceId=e0f92fe886bea789, duration=22 ms] 2025-04-08 20:27:43.019 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,ba875868acfb6683,7bf8ca4bb0efcba3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.11.12.102, traceId=ba875868acfb6683, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2025-04-08 20:27:43.019 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,ba875868acfb6683,7bf8ca4bb0efcba3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.11.12.102, traceId=ba875868acfb6683, duration=2 ms] 2025-04-08 20:27:43.079 INFO 3009027 --- [ocp-schedules-16,cea2597709ba1f2b,24d5e8ecb1d0ea9c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:27:43.079 INFO 3009027 --- [ocp-async-3,55803cdb480f428d,e76aa4398b973bc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:27:43.090 WARN 3009027 --- [ocp-async-5,b9bdcddc621fc3ca,273961401392d857] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:43.090 INFO 3009027 --- [ocp-async-4,36bc4c9c8578bbc1,ee1343b5ffa04ca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:43.117 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,90b6a76399ae4bb9,0ce3797f97b6cb63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.11.12.102, traceId=90b6a76399ae4bb9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-04-08 20:27:43.118 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,90b6a76399ae4bb9,0ce3797f97b6cb63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.11.12.102, traceId=90b6a76399ae4bb9, duration=6 ms] 2025-04-08 20:27:43.169 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:43.170 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:27:43.177719+08:00, lastProcessTime=2025-04-08T20:27:33.177646+08:00 2025-04-08 20:27:43.187 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:43.188 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743609600000000 2025-04-08 20:27:43.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:43.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743609600000000 values less than (1743609600000000)) 2025-04-08 20:27:43.388 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:27:43.177719+08:00 2025-04-08 20:27:43.405 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:27:43.408 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:27:43.408 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:27:43.879 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:43.881 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:43.898 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:43.899 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743696000000000 2025-04-08 20:27:43.899 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:43.900 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743696000000000 values less than (1743696000000000)) 2025-04-08 20:27:43.919 INFO 3009027 --- [ocp-schedules-7,57215d55cc1a4233,827a9f4f82411a74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:27:43.982 INFO 3009027 --- [ocp-schedules-3,acccc80f7bf08a4d,c7206e4e3bb87da5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:27:44.018 INFO 3009027 --- [ocp-schedules-3,acccc80f7bf08a4d,c7206e4e3bb87da5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:27:44.018 INFO 3009027 --- [ocp-schedules-3,acccc80f7bf08a4d,c7206e4e3bb87da5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:27:44.198 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:44.199 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:44.613 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:44.614 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:44.630 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:44.631 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743782400000000 2025-04-08 20:27:44.631 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:44.631 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743782400000000 values less than (1743782400000000)) 2025-04-08 20:27:45.087 INFO 3009027 --- [ocp-schedules-11,5e701097fc1586b0,c88d5aa815cbeb26] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:27:45.088 INFO 3009027 --- [ocp-schedules-11,5e701097fc1586b0,c88d5aa815cbeb26] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:27:45.090 INFO 3009027 --- [ocp-async-3,e3373447f30d6668,6edaabfb12432094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:45.090 WARN 3009027 --- [ocp-async-4,07c4720681b7dc18,8c8d16cd9e38ad80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:45.491 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,ec0e1048bd139289,64985dbc89214073] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=ec0e1048bd139289, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:27:45.491 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,ec0e1048bd139289,64985dbc89214073] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=ec0e1048bd139289, duration=4 ms] 2025-04-08 20:27:45.569 INFO 3009027 --- [prometheus-minute-schedule-1,9f5a6d0a8e374313,54e14ddcc4a06cb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:45.673 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,ee28cd79e364db73,1a4711f911352d7e] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-04-08 20:27:45.751 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1120, sql=alter table `ob_cluster_server_event` add partition(partition p1743782400000000 values less than (1743782400000000)) 2025-04-08 20:27:45.752 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:45.752 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:45.769 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:45.769 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743868800000000 2025-04-08 20:27:45.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:45.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743868800000000 values less than (1743868800000000)) 2025-04-08 20:27:45.878 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,ee28cd79e364db73,1a4711f911352d7e] c.c.s.h.OcpAuthenticationSuccessListener : User[name=admin, origin=LOCAL] authenticated successes. 2025-04-08 20:27:46.022 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,ee28cd79e364db73,1a4711f911352d7e] c.s.h.CustomAuthenticationSuccessHandler : Authentication successful for /api/v2/iam/login 2025-04-08 20:27:46.028 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,ee28cd79e364db73,1a4711f911352d7e] c.s.h.CustomAuthenticationSuccessHandler : Redirecting to saved url: /cluster/1/overview 2025-04-08 20:27:46.379 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,ce1d6d22554c890d,70d91319c87e60ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=ce1d6d22554c890d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:27:46.379 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,fcf70be15e562781,21957874a0caff2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=fcf70be15e562781, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:27:46.385 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,ce1d6d22554c890d,70d91319c87e60ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=ce1d6d22554c890d, duration=16 ms] 2025-04-08 20:27:46.401 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,fcf70be15e562781,21957874a0caff2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=fcf70be15e562781, duration=32 ms] 2025-04-08 20:27:46.405 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,86f946053317f25b,ec04f34e3f16d515] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.11.12.102, traceId=86f946053317f25b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-04-08 20:27:46.482 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,86f946053317f25b,ec04f34e3f16d515] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.11.12.102, traceId=86f946053317f25b, duration=113 ms] 2025-04-08 20:27:46.566 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:27:46.567 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:27:46.567 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:27:46.622 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,7a5ab2d73fce07bc,568ea0b6deb84614] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.11.12.102, traceId=7a5ab2d73fce07bc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2025-04-08 20:27:46.622 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,edb5dde090b33288,ba43145ffa595f73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.11.12.102, traceId=edb5dde090b33288, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2025-04-08 20:27:46.623 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,e8c1fb41cb39e48d,6aac7e33d78e2159] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/tenants/binlogService?id=1, client=10.11.12.102, traceId=e8c1fb41cb39e48d, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.listObClusterRelatedBinlogService(Long), args=1,] 2025-04-08 20:27:46.623 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,9f3195453e6a1b28,8153fd80162b56e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.11.12.102, traceId=9f3195453e6a1b28, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2025-04-08 20:27:46.625 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,fe7d0098e38b26ec,d1832a9f79a6985f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.11.12.102, traceId=fe7d0098e38b26ec, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2025-04-08 20:27:46.627 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,3fd0a519acbdd701,72ed35ec9b8c46c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2025-04-01T20%3A27%3A46%2B08%3A00&endTime=2025-04-08T20%3A27%3A46%2B08%3A00&page=1&size=5, client=10.11.12.102, traceId=3fd0a519acbdd701, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2025-04-01T20:27:46+08:00,2025-04-08T20:27:46+08:00,Page request [number: 0, size 5, sort: timestamp: DESC],] 2025-04-08 20:27:46.629 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-3,9f3195453e6a1b28,8153fd80162b56e9] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/summary client=10.11.12.102, traceId=9f3195453e6a1b28, args=1,, duration=56 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:46.630 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,edb5dde090b33288,ba43145ffa595f73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.11.12.102, traceId=edb5dde090b33288, duration=57 ms] 2025-04-08 20:27:46.632 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,7a5ab2d73fce07bc,568ea0b6deb84614] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/stats client=10.11.12.102, traceId=7a5ab2d73fce07bc, args=1,, duration=58 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:46.654 ERROR 3009027 --- [http-nio-0.0.0.0-8080-exec-6,fe7d0098e38b26ec,d1832a9f79a6985f] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.cluster.id.not.found, and args [1] 2025-04-08 20:27:46.655 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-6,fe7d0098e38b26ec,d1832a9f79a6985f] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/topSql/switch client=10.11.12.102, traceId=fe7d0098e38b26ec, args=1,, duration=41 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:46.667 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,e8c1fb41cb39e48d,6aac7e33d78e2159] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/tenants/binlogService client=10.11.12.102, traceId=e8c1fb41cb39e48d, duration=53 ms] 2025-04-08 20:27:46.709 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,3fd0a519acbdd701,72ed35ec9b8c46c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.11.12.102, traceId=3fd0a519acbdd701, duration=94 ms] 2025-04-08 20:27:46.722 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:46.723 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:46.725 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,8ff629886e945117,3644724775cb9f61] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=8ff629886e945117, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:27:46.739 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:46.739 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,8ff629886e945117,3644724775cb9f61] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=8ff629886e945117, duration=18 ms] 2025-04-08 20:27:46.739 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1743955200000000 2025-04-08 20:27:46.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:46.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1743955200000000 values less than (1743955200000000)) 2025-04-08 20:27:47.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4ccefee01a5a19ea,48915f3bdd47c93d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/stats?id=1, client=10.11.12.102, traceId=4ccefee01a5a19ea, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=1,] 2025-04-08 20:27:47.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,efd1b183c4aa17c7,55fccf4c142fe8e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/eventMeta?id=1, client=10.11.12.102, traceId=efd1b183c4aa17c7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=1,] 2025-04-08 20:27:47.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8b09c0603ebb55b1,96e92080e67db5fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.11.12.102, traceId=8b09c0603ebb55b1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-04-08 20:27:47.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,efd1b183c4aa17c7,55fccf4c142fe8e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/eventMeta client=10.11.12.102, traceId=efd1b183c4aa17c7, duration=6 ms] 2025-04-08 20:27:47.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,6593f93099330f7c,f27fc0239955834e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.11.12.102, traceId=6593f93099330f7c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2025-04-08 20:27:47.076 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,3171b06e94262cca,72e77cd57ae05439] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/tenants/binlogService?id=1, client=10.11.12.102, traceId=3171b06e94262cca, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.listObClusterRelatedBinlogService(Long), args=1,] 2025-04-08 20:27:47.077 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,6593f93099330f7c,f27fc0239955834e] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/summary client=10.11.12.102, traceId=6593f93099330f7c, args=1,, duration=8 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:47.077 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4ccefee01a5a19ea,48915f3bdd47c93d] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/stats client=10.11.12.102, traceId=4ccefee01a5a19ea, args=1,, duration=8 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:47.079 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,3171b06e94262cca,72e77cd57ae05439] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/tenants/binlogService client=10.11.12.102, traceId=3171b06e94262cca, duration=10 ms] 2025-04-08 20:27:47.080 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,736ba6aee833f4f9,a8271eb8831d4e57] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=736ba6aee833f4f9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:27:47.083 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8b09c0603ebb55b1,96e92080e67db5fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.11.12.102, traceId=8b09c0603ebb55b1, duration=14 ms] 2025-04-08 20:27:47.090 WARN 3009027 --- [ocp-async-6,c612d990c29ad07d,a8173dd4b2a59510] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:47.091 INFO 3009027 --- [ocp-async-3,6ada7bc35d466d40,d945d40f7bf712a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:47.196 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,1e02ce6cbfe7186c,dfa1b4391e79dfcb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/topSql/switch?id=1, client=10.11.12.102, traceId=1e02ce6cbfe7186c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=1,] 2025-04-08 20:27:47.197 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e744639800a9a1bd,e0f3142b0227de00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/summary?id=1, client=10.11.12.102, traceId=e744639800a9a1bd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=1,] 2025-04-08 20:27:47.198 ERROR 3009027 --- [http-nio-0.0.0.0-8080-exec-10,1e02ce6cbfe7186c,dfa1b4391e79dfcb] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.cluster.id.not.found, and args [1] 2025-04-08 20:27:47.198 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-10,1e02ce6cbfe7186c,dfa1b4391e79dfcb] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/topSql/switch client=10.11.12.102, traceId=1e02ce6cbfe7186c, args=1,, duration=6 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:47.198 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e744639800a9a1bd,e0f3142b0227de00] c.o.o.s.c.trace.RequestTracingAspect : API Error: [GET /api/v2/ob/clusters/1/summary client=10.11.12.102, traceId=e744639800a9a1bd, args=1,, duration=7 ms, errorMsg=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1] 2025-04-08 20:27:47.198 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,4d18b4dff043f269,009fb72d1163c013] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/1/events?startTime=2025-04-01T20%3A27%3A46%2B08%3A00&endTime=2025-04-08T20%3A27%3A46%2B08%3A00&page=1&size=5, client=10.11.12.102, traceId=4d18b4dff043f269, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=1,2025-04-01T20:27:46+08:00,2025-04-08T20:27:46+08:00,Page request [number: 0, size 5, sort: timestamp: DESC],] 2025-04-08 20:27:47.199 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:47.200 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:47.201 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,2fe77b71b84c1654,493618857414f806] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=1&status=RUNNING%2CFAILED, client=10.11.12.102, traceId=2fe77b71b84c1654, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,1,] 2025-04-08 20:27:47.217 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:47.220 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,2fe77b71b84c1654,493618857414f806] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.11.12.102, traceId=2fe77b71b84c1654, duration=28 ms] 2025-04-08 20:27:47.227 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,4d18b4dff043f269,009fb72d1163c013] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/1/events client=10.11.12.102, traceId=4d18b4dff043f269, duration=35 ms] 2025-04-08 20:27:47.233 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,736ba6aee833f4f9,a8271eb8831d4e57] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=736ba6aee833f4f9, duration=164 ms] 2025-04-08 20:27:47.234 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:47.235 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1744041600000000 2025-04-08 20:27:47.235 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.235 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1744041600000000 values less than (1744041600000000)) 2025-04-08 20:27:47.501 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.502 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:47.519 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:47.520 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1744128000000000 2025-04-08 20:27:47.520 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.520 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1744128000000000 values less than (1744128000000000)) 2025-04-08 20:27:47.720 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.721 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:47.739 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:47.740 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1744214400000000 2025-04-08 20:27:47.740 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.741 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1744214400000000 values less than (1744214400000000)) 2025-04-08 20:27:47.954 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.955 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:47.971 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:47.972 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_cluster_server_event, partition=p1744300800000000 2025-04-08 20:27:47.972 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:47.973 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_cluster_server_event` add partition(partition p1744300800000000 values less than (1744300800000000)) 2025-04-08 20:27:48.114 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,6f493d2fad7bb432,68ae21d93ac902a7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=6f493d2fad7bb432, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:27:48.114 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,6f493d2fad7bb432,68ae21d93ac902a7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=6f493d2fad7bb432, duration=3 ms] 2025-04-08 20:27:48.165 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.166 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_cluster_server_event' and table_type='BASE TABLE' 2025-04-08 20:27:48.182 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_cluster_server_event 2025-04-08 20:27:48.184 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.184 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:48.184 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.184 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 20:27:48.187 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743436800 2025-04-08 20:27:48.187 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743436800 values less than (1743436800)) 2025-04-08 20:27:48.359 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d070cd0bdb53452b,2ce16f7973b1432a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=d070cd0bdb53452b, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:27:48.359 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d070cd0bdb53452b,2ce16f7973b1432a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=d070cd0bdb53452b, duration=3 ms] 2025-04-08 20:27:48.359 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,77dc4b91fa891646,12076b1e8f75947b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=77dc4b91fa891646, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:27:48.359 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,77dc4b91fa891646,12076b1e8f75947b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=77dc4b91fa891646, duration=3 ms] 2025-04-08 20:27:48.360 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,92979ce01c158f62,5d83626cfc0ae82e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=92979ce01c158f62, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:27:48.360 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,92979ce01c158f62,5d83626cfc0ae82e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=92979ce01c158f62, duration=4 ms] 2025-04-08 20:27:48.534 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:27:48.534 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:27:48.534 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:27:48.547 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.548 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:48.564 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:48.565 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743523200 2025-04-08 20:27:48.565 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.566 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743523200 values less than (1743523200)) 2025-04-08 20:27:48.569 INFO 3009027 --- [prometheus-minute-schedule-1,9d810c2e9c3b638f,016c4cf214a88279] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:48.697 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.698 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:48.714 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:48.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743609600 2025-04-08 20:27:48.715 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.716 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743609600 values less than (1743609600)) 2025-04-08 20:27:48.854 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:48.872 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:48.872 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743696000 2025-04-08 20:27:48.873 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:48.873 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743696000 values less than (1743696000)) 2025-04-08 20:27:49.020 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.021 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:49.036 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:49.037 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743782400 2025-04-08 20:27:49.037 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743782400 values less than (1743782400)) 2025-04-08 20:27:49.091 WARN 3009027 --- [ocp-async-2,c9315f3e24fbd2af,22f358805052baee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:49.091 INFO 3009027 --- [ocp-async-3,45046b9a7743d6e7,c01cefce95977f1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:49.351 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.352 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:49.368 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:49.368 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743868800 2025-04-08 20:27:49.369 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.369 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743868800 values less than (1743868800)) 2025-04-08 20:27:49.523 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.524 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:49.541 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:49.541 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1743955200 2025-04-08 20:27:49.542 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.542 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1743955200 values less than (1743955200)) 2025-04-08 20:27:49.850 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.851 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:49.867 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:49.867 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1744041600 2025-04-08 20:27:49.868 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:49.868 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1744041600 values less than (1744041600)) 2025-04-08 20:27:50.085 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.086 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:50.102 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:50.103 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1744128000 2025-04-08 20:27:50.103 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1744128000 values less than (1744128000)) 2025-04-08 20:27:50.200 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:50.201 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:50.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.224 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:50.240 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:50.241 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1744214400 2025-04-08 20:27:50.241 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.241 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1744214400 values less than (1744214400)) 2025-04-08 20:27:50.466 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.466 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:50.482 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:50.483 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_1, partition=p1744300800 2025-04-08 20:27:50.483 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.483 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_1` add partition(partition p1744300800 values less than (1744300800)) 2025-04-08 20:27:50.580 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.581 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_1' and table_type='BASE TABLE' 2025-04-08 20:27:50.597 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_1 2025-04-08 20:27:50.598 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.598 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:50.598 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.598 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 20:27:50.601 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741449600 2025-04-08 20:27:50.602 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.602 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741449600 values less than (1741449600)) 2025-04-08 20:27:50.750 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.751 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:50.767 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:50.768 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741536000 2025-04-08 20:27:50.768 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:50.768 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741536000 values less than (1741536000)) 2025-04-08 20:27:51.034 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.035 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:51.051 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:51.052 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741622400 2025-04-08 20:27:51.052 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.052 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741622400 values less than (1741622400)) 2025-04-08 20:27:51.091 WARN 3009027 --- [ocp-async-4,7c8587a12a56ddc1,4b5e1e27671ac531] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:51.091 INFO 3009027 --- [ocp-async-3,acae1572d02a0d24,c6fdf036b7e44350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:51.219 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.220 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:51.236 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:51.237 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741708800 2025-04-08 20:27:51.237 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.238 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741708800 values less than (1741708800)) 2025-04-08 20:27:51.356 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,19a3f51d0264be1f,ed6abcdac1e521e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=19a3f51d0264be1f, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:27:51.364 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.365 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:51.368 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,19a3f51d0264be1f,ed6abcdac1e521e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=19a3f51d0264be1f, duration=16 ms] 2025-04-08 20:27:51.383 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:51.384 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741795200 2025-04-08 20:27:51.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.385 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741795200 values less than (1741795200)) 2025-04-08 20:27:51.569 INFO 3009027 --- [prometheus-minute-schedule-1,8ecbd63f2d66d1b5,60e381f96b453f51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:51.654 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:51.671 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:51.671 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741881600 2025-04-08 20:27:51.672 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:51.672 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741881600 values less than (1741881600)) 2025-04-08 20:27:52.165 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:52.166 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:52.181 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:52.182 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1741968000 2025-04-08 20:27:52.182 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:52.183 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1741968000 values less than (1741968000)) 2025-04-08 20:27:52.729 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:52.730 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:52.747 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:52.748 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742054400 2025-04-08 20:27:52.748 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:52.748 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742054400 values less than (1742054400)) 2025-04-08 20:27:52.995 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,8f44cadfb2e0a36c,96925bef5d623622] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /?null, client=10.11.12.102, traceId=8f44cadfb2e0a36c, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.WebController.index(), args=] 2025-04-08 20:27:52.995 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,8f44cadfb2e0a36c,96925bef5d623622] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET / client=10.11.12.102, traceId=8f44cadfb2e0a36c, duration=4 ms] 2025-04-08 20:27:53.057 INFO 3009027 --- [ocp-schedules-2,935c5c6303326a61,7fc7f56e56074414] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:27:53.057 INFO 3009027 --- [ocp-schedules-13,e0656ed62e678e2f,a5e0b009c1931867] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:27:53.058 INFO 3009027 --- [ocp-schedules-16,3aa67c506f350d33,45ebc31118b450d4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:27:53.058 INFO 3009027 --- [ocp-schedules-17,8699e15b05648031,b79a9226997fa5ad] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:27:53.060 INFO 3009027 --- [ocp-schedules-25,489b99d2cde7ccf3,8766db70fa565d34] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:27:53.060 INFO 3009027 --- [ocp-schedules-7,2dc015cb2a9f7947,a9fa92b6171b6e4c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:27:53.060 INFO 3009027 --- [ocp-schedules-25,489b99d2cde7ccf3,8766db70fa565d34] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:27:53.060 INFO 3009027 --- [ocp-schedules-7,2dc015cb2a9f7947,a9fa92b6171b6e4c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:27:53.060 INFO 3009027 --- [ocp-schedules-18,a156465666159757,e651ff60a2dcae99] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:27:53.060 INFO 3009027 --- [ocp-schedules-18,a156465666159757,e651ff60a2dcae99] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:27:53.079 INFO 3009027 --- [ocp-schedules-22,68988156cd34528c,66712e65cca41420] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:27:53.079 INFO 3009027 --- [ocp-async-3,e049e1a3e0e07c7c,5cd6ccfe729da6a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:27:53.091 WARN 3009027 --- [ocp-async-5,385905db0b74bf32,b261da667eaebcd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:53.091 INFO 3009027 --- [ocp-async-6,0617822fe012d4b8,ca26bed5656211d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:53.109 INFO 3009027 --- [ocp-schedules-4,e7fe9b68dcdc03b4,e3d20c70ea6eee3e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:27:53.110 INFO 3009027 --- [ocp-schedules-5,a9dacfd4e9d8c60e,bb36a7de94560f4b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:27:53.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:27:53.176497+08:00, lastProcessTime=2025-04-08T20:27:43.177719+08:00 2025-04-08 20:27:53.201 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:53.201 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:53.379 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:53.380 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:53.396 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:53.397 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742140800 2025-04-08 20:27:53.397 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:53.397 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742140800 values less than (1742140800)) 2025-04-08 20:27:53.407 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,a92e15a4b871d2c6,18da784ff65c3143] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.11.12.102, traceId=a92e15a4b871d2c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-04-08 20:27:53.408 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,a92e15a4b871d2c6,18da784ff65c3143] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.11.12.102, traceId=a92e15a4b871d2c6, duration=4 ms] 2025-04-08 20:27:53.439 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:27:53.176497+08:00 2025-04-08 20:27:53.525 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,5f708985e0586746,bef6cafab4792ee7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=5f708985e0586746, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:27:53.525 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,dcfb7774145be63d,051b36de8366983a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=dcfb7774145be63d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:27:53.525 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,dcfb7774145be63d,051b36de8366983a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=dcfb7774145be63d, duration=3 ms] 2025-04-08 20:27:53.531 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,5f708985e0586746,bef6cafab4792ee7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=5f708985e0586746, duration=9 ms] 2025-04-08 20:27:53.720 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d83067ee410eca4d,32911578caa9adf9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=d83067ee410eca4d, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:27:53.722 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:27:53.722 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:27:53.722 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:27:53.724 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d83067ee410eca4d,32911578caa9adf9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=d83067ee410eca4d, duration=49 ms] 2025-04-08 20:27:53.796 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,061db09c0723c370,10e3fc6af9434eba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=061db09c0723c370, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:27:53.890 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,061db09c0723c370,10e3fc6af9434eba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=061db09c0723c370, duration=99 ms] 2025-04-08 20:27:53.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:53.913 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:53.929 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:53.930 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742227200 2025-04-08 20:27:53.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:53.931 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742227200 values less than (1742227200)) 2025-04-08 20:27:53.989 INFO 3009027 --- [ocp-schedules-20,9a7b8e4a8c222a8c,d1c3e509e215d9bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:27:54.123 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,37953bcca8e829d2,926159d89085591a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.11.12.102, traceId=37953bcca8e829d2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2025-04-08 20:27:54.123 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,3b373a2534490424,132fa7616f68d408] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.11.12.102, traceId=3b373a2534490424, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2025-04-08 20:27:54.124 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,907b0c764c11bed6,f520760123188302] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=907b0c764c11bed6, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:27:54.124 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,26875383e399f41f,521bff8870dbd3eb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.11.12.102, traceId=26875383e399f41f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2025-04-08 20:27:54.125 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,d1e7999148bad400,cd4ebac6d5d883a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.11.12.102, traceId=d1e7999148bad400, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2025-04-08 20:27:54.126 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,4ee02c04b601ede3,0ec2575387d57514] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=4ee02c04b601ede3, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:27:54.126 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,4ee02c04b601ede3,0ec2575387d57514] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=4ee02c04b601ede3, duration=9 ms] 2025-04-08 20:27:54.131 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,907b0c764c11bed6,f520760123188302] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=907b0c764c11bed6, duration=14 ms] 2025-04-08 20:27:54.151 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,5577bd0f7709614b,a633490851d16c00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP, client=10.11.12.102, traceId=5577bd0f7709614b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,OBCLUSTER,] 2025-04-08 20:27:54.151 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,994082387e438c61,8c319bf323d7b891] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP, client=10.11.12.102, traceId=994082387e438c61, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,OBCLUSTER,] 2025-04-08 20:27:54.171 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,26875383e399f41f,521bff8870dbd3eb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.11.12.102, traceId=26875383e399f41f, duration=53 ms] 2025-04-08 20:27:54.172 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,d1e7999148bad400,cd4ebac6d5d883a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.11.12.102, traceId=d1e7999148bad400, duration=53 ms] 2025-04-08 20:27:54.175 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,3b373a2534490424,132fa7616f68d408] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.11.12.102, traceId=3b373a2534490424, duration=56 ms] 2025-04-08 20:27:54.182 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,37953bcca8e829d2,926159d89085591a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.11.12.102, traceId=37953bcca8e829d2, duration=63 ms] 2025-04-08 20:27:54.210 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:54.211 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:54.227 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:54.228 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742313600 2025-04-08 20:27:54.228 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:54.228 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742313600 values less than (1742313600)) 2025-04-08 20:27:54.242 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,71a9e6e19db582f7,fcbe673702569374] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP, client=10.11.12.102, traceId=71a9e6e19db582f7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,OBCLUSTER,] 2025-04-08 20:27:54.244 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,41d9ae7fd077d2e2,4d2e438aed4f8a39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.11.12.102, traceId=41d9ae7fd077d2e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-04-08 20:27:54.251 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,41d9ae7fd077d2e2,4d2e438aed4f8a39] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_cpu_assigned_percentage, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 60 2025-04-08 20:27:54.257 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,41d9ae7fd077d2e2,4d2e438aed4f8a39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.11.12.102, traceId=41d9ae7fd077d2e2, duration=19 ms] 2025-04-08 20:27:54.293 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8ea7a76556408d32,1410785a67f2b198] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.11.12.102, traceId=8ea7a76556408d32, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-04-08 20:27:54.297 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8ea7a76556408d32,1410785a67f2b198] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_disk_used_percentage, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 60 2025-04-08 20:27:54.298 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,8ea7a76556408d32,1410785a67f2b198] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.11.12.102, traceId=8ea7a76556408d32, duration=9 ms] 2025-04-08 20:27:54.393 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5a2eafba50f03672,191fbaa253fd1b4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.11.12.102, traceId=5a2eafba50f03672, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-04-08 20:27:54.394 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,89e7aff7ead1d210,a3e2149e5fca6f04] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.11.12.102, traceId=89e7aff7ead1d210, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-04-08 20:27:54.395 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,71a9e6e19db582f7,fcbe673702569374] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=71a9e6e19db582f7, duration=157 ms] 2025-04-08 20:27:54.395 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,994082387e438c61,8c319bf323d7b891] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=994082387e438c61, duration=249 ms] 2025-04-08 20:27:54.395 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,5577bd0f7709614b,a633490851d16c00] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=5577bd0f7709614b, duration=249 ms] 2025-04-08 20:27:54.395 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5a2eafba50f03672,191fbaa253fd1b4c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_memory_assigned_percentage, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 60 2025-04-08 20:27:54.399 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5a2eafba50f03672,191fbaa253fd1b4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.11.12.102, traceId=5a2eafba50f03672, duration=12 ms] 2025-04-08 20:27:54.400 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,89e7aff7ead1d210,a3e2149e5fca6f04] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_memory_percent, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-04-08 20:27:54.404 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,89e7aff7ead1d210,a3e2149e5fca6f04] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.11.12.102, traceId=89e7aff7ead1d210, duration=14 ms] 2025-04-08 20:27:54.454 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e1f7e814a9e1e247,3c6f9e88be5bd9e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.11.12.102, traceId=e1f7e814a9e1e247, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-04-08 20:27:54.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,eaba112a5ce54e4b,4ee82475c622cb52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.11.12.102, traceId=eaba112a5ce54e4b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-04-08 20:27:54.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,a50ce6c9f3ae7fb9,6cfa0faf1da56273] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.11.12.102, traceId=a50ce6c9f3ae7fb9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-04-08 20:27:54.457 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e1f7e814a9e1e247,3c6f9e88be5bd9e1] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_tenant_disk_usage_v4, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-04-08 20:27:54.457 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-6,a50ce6c9f3ae7fb9,6cfa0faf1da56273] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_cpu_percent, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-04-08 20:27:54.458 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e1f7e814a9e1e247,3c6f9e88be5bd9e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.11.12.102, traceId=e1f7e814a9e1e247, duration=8 ms] 2025-04-08 20:27:54.458 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,a50ce6c9f3ae7fb9,6cfa0faf1da56273] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.11.12.102, traceId=a50ce6c9f3ae7fb9, duration=8 ms] 2025-04-08 20:27:54.460 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,eaba112a5ce54e4b,4ee82475c622cb52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.11.12.102, traceId=eaba112a5ce54e4b, duration=9 ms] 2025-04-08 20:27:54.556 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:54.557 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:54.569 INFO 3009027 --- [prometheus-minute-schedule-1,ab618434ed590ab4,c5568bf976945527] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:54.572 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:54.573 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742400000 2025-04-08 20:27:54.573 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:54.574 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742400000 values less than (1742400000)) 2025-04-08 20:27:54.788 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:54.789 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:54.804 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:54.805 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742486400 2025-04-08 20:27:54.805 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:54.805 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742486400 values less than (1742486400)) 2025-04-08 20:27:54.929 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,7965808fe4030bfe,82a180796a3d2fd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=7965808fe4030bfe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:27:54.932 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,7965808fe4030bfe,82a180796a3d2fd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=7965808fe4030bfe, duration=6 ms] 2025-04-08 20:27:55.042 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.043 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:55.061 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:55.062 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742572800 2025-04-08 20:27:55.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742572800 values less than (1742572800)) 2025-04-08 20:27:55.091 INFO 3009027 --- [ocp-schedules-8,04a5d1e81d8d6a84,78b12971f6f2c62a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:27:55.091 WARN 3009027 --- [ocp-async-9,4b1b60a80cb21192,85b34f1eac14ded3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:55.091 INFO 3009027 --- [ocp-async-4,9f6502b9410c763e,d3875f8fb305261d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:55.092 INFO 3009027 --- [ocp-schedules-8,04a5d1e81d8d6a84,78b12971f6f2c62a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:27:55.238 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.239 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:55.258 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:55.259 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742659200 2025-04-08 20:27:55.259 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.259 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742659200 values less than (1742659200)) 2025-04-08 20:27:55.414 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.415 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:55.430 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:55.431 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742745600 2025-04-08 20:27:55.431 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.432 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742745600 values less than (1742745600)) 2025-04-08 20:27:55.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,2ce56324e8601049,8c113662ae05f130] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=2ce56324e8601049, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:27:55.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,2ce56324e8601049,8c113662ae05f130] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=2ce56324e8601049, duration=4 ms] 2025-04-08 20:27:55.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,fede40ecb5bbaf50,47d26d3ce68f26f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=fede40ecb5bbaf50, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:27:55.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,12e9c33f14be776d,21d7aae5bff685a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=12e9c33f14be776d, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:27:55.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,12e9c33f14be776d,21d7aae5bff685a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=12e9c33f14be776d, duration=4 ms] 2025-04-08 20:27:55.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,fede40ecb5bbaf50,47d26d3ce68f26f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=fede40ecb5bbaf50, duration=4 ms] 2025-04-08 20:27:55.600 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:27:55.600 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:27:55.600 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:27:55.613 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.614 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:55.631 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:55.631 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742832000 2025-04-08 20:27:55.632 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.632 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742832000 values less than (1742832000)) 2025-04-08 20:27:55.742 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.743 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:55.760 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:55.760 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1742918400 2025-04-08 20:27:55.761 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.761 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1742918400 values less than (1742918400)) 2025-04-08 20:27:55.805 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.806 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:55.821 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:55.822 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743004800 2025-04-08 20:27:55.822 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:55.823 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743004800 values less than (1743004800)) 2025-04-08 20:27:55.837 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,c350cb292194f592,57ba6faaba340b36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=c350cb292194f592, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:27:55.837 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1a778a8d923faac7,c42477070a9f2513] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=1a778a8d923faac7, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:27:55.837 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,c350cb292194f592,57ba6faaba340b36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=c350cb292194f592, duration=3 ms] 2025-04-08 20:27:55.838 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,cb75c74f4132d119,dd17ad77ecdb4c36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=cb75c74f4132d119, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:27:55.838 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,74ca0b13a5543258,f003d61994491ba1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=74ca0b13a5543258, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:27:55.838 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,cb75c74f4132d119,dd17ad77ecdb4c36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=cb75c74f4132d119, duration=4 ms] 2025-04-08 20:27:55.841 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1a778a8d923faac7,c42477070a9f2513] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=1a778a8d923faac7, duration=7 ms] 2025-04-08 20:27:55.843 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,74ca0b13a5543258,f003d61994491ba1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=74ca0b13a5543258, duration=9 ms] 2025-04-08 20:27:55.901 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,6251576cbee5f91d,be5ad155c6512ac7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=6251576cbee5f91d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:27:55.909 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,6251576cbee5f91d,be5ad155c6512ac7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=6251576cbee5f91d, duration=56 ms] 2025-04-08 20:27:56.018 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,79ee76798b516f70,458746d3f9c5d8d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A27%3A56%2B08%3A00&endTime=2025-04-08T20%3A27%3A56%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=79ee76798b516f70, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:27:56+08:00,2025-04-08T20:27:56+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:27:56.018 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,79ee76798b516f70,458746d3f9c5d8d2] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 30 2025-04-08 20:27:56.023 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,79ee76798b516f70,458746d3f9c5d8d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=79ee76798b516f70, duration=11 ms] 2025-04-08 20:27:56.039 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.040 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:56.056 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:56.057 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743091200 2025-04-08 20:27:56.057 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.058 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743091200 values less than (1743091200)) 2025-04-08 20:27:56.201 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:56.202 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:56.230 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.231 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:56.248 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:56.248 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743177600 2025-04-08 20:27:56.249 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.249 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743177600 values less than (1743177600)) 2025-04-08 20:27:56.438 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.439 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:56.455 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:56.456 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743264000 2025-04-08 20:27:56.456 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.457 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743264000 values less than (1743264000)) 2025-04-08 20:27:56.586 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.587 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:56.604 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:56.604 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743350400 2025-04-08 20:27:56.605 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.605 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743350400 values less than (1743350400)) 2025-04-08 20:27:56.713 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.714 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:56.729 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:56.730 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743436800 2025-04-08 20:27:56.730 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:56.730 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743436800 values less than (1743436800)) 2025-04-08 20:27:56.758 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d02bbc79ec1b3023,fe86b0f2b2a5b6c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=d02bbc79ec1b3023, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:27:56.761 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d02bbc79ec1b3023,fe86b0f2b2a5b6c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=d02bbc79ec1b3023, duration=6 ms] 2025-04-08 20:27:57.086 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.087 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:57.091 WARN 3009027 --- [ocp-async-6,c0be0941717b88ac,0e283cfd94b72c12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:57.091 INFO 3009027 --- [ocp-async-4,992909806c588361,0ee71c98004910e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:57.103 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:57.104 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743523200 2025-04-08 20:27:57.104 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.105 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743523200 values less than (1743523200)) 2025-04-08 20:27:57.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.342 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:57.358 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:57.358 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743609600 2025-04-08 20:27:57.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.359 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743609600 values less than (1743609600)) 2025-04-08 20:27:57.569 INFO 3009027 --- [prometheus-minute-schedule-1,7f3eb914a57d2c3a,8e6e9f86a233051d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115280, interval=60 2025-04-08 20:27:57.624 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.625 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:57.642 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:57.642 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743696000 2025-04-08 20:27:57.643 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.643 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743696000 values less than (1743696000)) 2025-04-08 20:27:57.891 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.892 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:57.908 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:57.909 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743782400 2025-04-08 20:27:57.909 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.910 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743782400 values less than (1743782400)) 2025-04-08 20:27:57.953 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.954 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:57.970 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:57.970 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743868800 2025-04-08 20:27:57.971 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:57.971 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743868800 values less than (1743868800)) 2025-04-08 20:27:58.207 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.208 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:58.228 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:58.229 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1743955200 2025-04-08 20:27:58.229 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.230 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1743955200 values less than (1743955200)) 2025-04-08 20:27:58.404 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a7616cf80c9eefaf,cd439c0fa14b7f4e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=a7616cf80c9eefaf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:27:58.404 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a7616cf80c9eefaf,cd439c0fa14b7f4e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=a7616cf80c9eefaf, duration=3 ms] 2025-04-08 20:27:58.444 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.445 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:58.461 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:58.461 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1744041600 2025-04-08 20:27:58.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.462 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1744041600 values less than (1744041600)) 2025-04-08 20:27:58.504 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,dafccdd953e5a164,77758e8db20e3419] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=dafccdd953e5a164, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:27:58.506 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,dafccdd953e5a164,77758e8db20e3419] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=dafccdd953e5a164, duration=5 ms] 2025-04-08 20:27:58.570 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.571 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:58.587 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:58.588 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1744128000 2025-04-08 20:27:58.589 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.589 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1744128000 values less than (1744128000)) 2025-04-08 20:27:58.636 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:58.652 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:58.653 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1744214400 2025-04-08 20:27:58.653 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.654 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1744214400 values less than (1744214400)) 2025-04-08 20:27:58.801 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.802 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:58.817 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:58.818 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ocp_metric_data_60, partition=p1744300800 2025-04-08 20:27:58.818 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.819 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ocp_metric_data_60` add partition(partition p1744300800 values less than (1744300800)) 2025-04-08 20:27:58.926 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.927 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ocp_metric_data_60' and table_type='BASE TABLE' 2025-04-08 20:27:58.945 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ocp_metric_data_60 2025-04-08 20:27:58.946 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.946 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:58.946 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.946 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 20:27:58.951 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2022 2025-04-08 20:27:58.951 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:58.951 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2022 values less than (1672531200)) 2025-04-08 20:27:59.091 WARN 3009027 --- [ocp-async-5,54bf33fbeb29729c,d103bdb2c8432d23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:27:59.091 INFO 3009027 --- [ocp-async-1,8a515c5fbe1cc607,df9fbae7fa3701b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:27:59.120 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.121 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2025-04-08 20:27:59.138 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2025-04-08 20:27:59.139 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2023 2025-04-08 20:27:59.139 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.139 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2023 values less than (1704067200)) 2025-04-08 20:27:59.202 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:27:59.203 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:27:59.398 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.399 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2025-04-08 20:27:59.418 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2025-04-08 20:27:59.419 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2024 2025-04-08 20:27:59.420 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.420 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2024 values less than (1735689600)) 2025-04-08 20:27:59.653 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.654 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2025-04-08 20:27:59.671 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2025-04-08 20:27:59.672 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2025 2025-04-08 20:27:59.672 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.672 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2025 values less than (1767225600)) 2025-04-08 20:27:59.798 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.799 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2025-04-08 20:27:59.815 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2025-04-08 20:27:59.816 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_hour_data, partition=p2026 2025-04-08 20:27:59.816 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.817 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_hour_data` add partition(partition p2026 values less than (1798761600)) 2025-04-08 20:27:59.952 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.953 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_hour_data' and table_type='BASE TABLE' 2025-04-08 20:27:59.969 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_hour_data 2025-04-08 20:27:59.970 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.970 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:27:59.970 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.971 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 20:27:59.973 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2022 2025-04-08 20:27:59.974 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:27:59.974 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2022 values less than (1672531200)) 2025-04-08 20:28:00.068 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.069 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2025-04-08 20:28:00.084 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2025-04-08 20:28:00.085 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2023 2025-04-08 20:28:00.085 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.086 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2023 values less than (1704067200)) 2025-04-08 20:28:00.098 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,db45ac85f025b69e,1e1d8620448e0aec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.11.12.102, traceId=db45ac85f025b69e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,OBCLUSTER,] 2025-04-08 20:28:00.103 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,db45ac85f025b69e,1e1d8620448e0aec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=db45ac85f025b69e, duration=9 ms] 2025-04-08 20:28:00.119 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,78e407ded917af02,9e93c7e43e263449] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.11.12.102, traceId=78e407ded917af02, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:28:00.152 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,78e407ded917af02,9e93c7e43e263449] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.11.12.102, traceId=78e407ded917af02, duration=39 ms] 2025-04-08 20:28:00.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.359 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2025-04-08 20:28:00.375 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2025-04-08 20:28:00.375 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2024 2025-04-08 20:28:00.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.376 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2024 values less than (1735689600)) 2025-04-08 20:28:00.569 INFO 3009027 --- [prometheus-minute-schedule-1,3587550afdf78037,49a653d1b58a09be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:00.578 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,65abb4a0623f2abb,268667d438371e9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A00%2B08%3A00&endTime=2025-04-08T20%3A28%3A00%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=65abb4a0623f2abb, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28+08:00,2025-04-08T20:28+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:00.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.578 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,65abb4a0623f2abb,268667d438371e9a] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = tps, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:00.579 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2025-04-08 20:28:00.580 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,65abb4a0623f2abb,268667d438371e9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=65abb4a0623f2abb, duration=6 ms] 2025-04-08 20:28:00.585 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,94f1162105fdf972,e6c8692f2327149b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A00%2B08%3A00&endTime=2025-04-08T20%3A28%3A00%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=94f1162105fdf972, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28+08:00,2025-04-08T20:28+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:00.585 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-7,94f1162105fdf972,e6c8692f2327149b] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:00.587 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,94f1162105fdf972,e6c8692f2327149b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=94f1162105fdf972, duration=5 ms] 2025-04-08 20:28:00.598 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2025-04-08 20:28:00.599 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2025 2025-04-08 20:28:00.599 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.599 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2025 values less than (1767225600)) 2025-04-08 20:28:00.607 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,10c4c257945b527f,fbbc5043f172836c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A00%2B08%3A00&endTime=2025-04-08T20%3A28%3A00%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=10c4c257945b527f, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28+08:00,2025-04-08T20:28+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:00.607 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-8,10c4c257945b527f,fbbc5043f172836c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = tps_rt, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:00.608 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,10c4c257945b527f,fbbc5043f172836c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=10c4c257945b527f, duration=5 ms] 2025-04-08 20:28:00.619 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,971597b28b209970,ace6adc0cdd6955b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A00%2B08%3A00&endTime=2025-04-08T20%3A28%3A00%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=971597b28b209970, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28+08:00,2025-04-08T20:28+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:00.620 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,971597b28b209970,ace6adc0cdd6955b] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_rt, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:00.622 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,971597b28b209970,ace6adc0cdd6955b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=971597b28b209970, duration=6 ms] 2025-04-08 20:28:00.750 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.751 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2025-04-08 20:28:00.769 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2025-04-08 20:28:00.769 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_daily_data, partition=p2026 2025-04-08 20:28:00.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_daily_data` add partition(partition p2026 values less than (1798761600)) 2025-04-08 20:28:00.911 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_daily_data' and table_type='BASE TABLE' 2025-04-08 20:28:00.929 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_daily_data 2025-04-08 20:28:00.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.930 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:00.931 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.931 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 20:28:00.934 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250401 2025-04-08 20:28:00.935 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:00.935 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250401 values less than (1743436800)) 2025-04-08 20:28:01.032 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,e4b362c18d9d0c85,8b68140b7455ec30] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=e4b362c18d9d0c85, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:01.035 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,e4b362c18d9d0c85,8b68140b7455ec30] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=e4b362c18d9d0c85, duration=7 ms] 2025-04-08 20:28:01.088 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.089 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:01.092 INFO 3009027 --- [ocp-async-8,45b346d0b800ab15,4fa0097fbd24edd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:01.092 WARN 3009027 --- [ocp-async-9,3bcb7661866a1645,4bbe0da7fde18f9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:01.105 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:01.105 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250402 2025-04-08 20:28:01.106 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.106 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250402 values less than (1743523200)) 2025-04-08 20:28:01.290 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.291 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:01.307 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:01.308 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250403 2025-04-08 20:28:01.309 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.309 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250403 values less than (1743609600)) 2025-04-08 20:28:01.507 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.508 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:01.526 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:01.527 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250404 2025-04-08 20:28:01.527 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.527 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250404 values less than (1743696000)) 2025-04-08 20:28:01.687 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,35a3947781e74bf9,1ad72247ac7c572a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=35a3947781e74bf9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:28:01.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,65abd29539727cef,376e46ede0266b39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=65abd29539727cef, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:28:01.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,be21f1a65acb1af4,185361d2a61f5985] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A28%3A01%2B08%3A00&endTime=2025-04-08T20%3A28%3A01%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=be21f1a65acb1af4, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:01+08:00,2025-04-08T20:28:01+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:28:01.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,3093bb2024a6a4d4,226056ae8065c243] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=3093bb2024a6a4d4, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:28:01.688 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,be21f1a65acb1af4,185361d2a61f5985] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 30 2025-04-08 20:28:01.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,1b8817ff220f23ab,7ab9a8ad44057987] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=1b8817ff220f23ab, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:28:01.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,65abd29539727cef,376e46ede0266b39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=65abd29539727cef, duration=3 ms] 2025-04-08 20:28:01.689 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,3093bb2024a6a4d4,226056ae8065c243] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=3093bb2024a6a4d4, duration=4 ms] 2025-04-08 20:28:01.689 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,35a3947781e74bf9,1ad72247ac7c572a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=35a3947781e74bf9, duration=6 ms] 2025-04-08 20:28:01.690 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,be21f1a65acb1af4,185361d2a61f5985] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=be21f1a65acb1af4, duration=5 ms] 2025-04-08 20:28:01.691 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,1b8817ff220f23ab,7ab9a8ad44057987] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=1b8817ff220f23ab, duration=7 ms] 2025-04-08 20:28:01.828 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,bcce16e5007e9d67,274e9046fe80b388] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=bcce16e5007e9d67, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:28:01.833 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,bcce16e5007e9d67,274e9046fe80b388] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=bcce16e5007e9d67, duration=129 ms] 2025-04-08 20:28:01.850 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.851 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:01.869 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:01.870 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250405 2025-04-08 20:28:01.870 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:01.871 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250405 values less than (1743782400)) 2025-04-08 20:28:02.203 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:02.204 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:02.333 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:02.334 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:02.349 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:02.350 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250406 2025-04-08 20:28:02.350 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:02.351 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250406 values less than (1743868800)) 2025-04-08 20:28:02.662 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,bff02b2f113d4e80,7024e93711b3f574] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=bff02b2f113d4e80, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:02.665 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,bff02b2f113d4e80,7024e93711b3f574] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=bff02b2f113d4e80, duration=6 ms] 2025-04-08 20:28:02.821 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:02.822 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:02.839 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:02.839 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250407 2025-04-08 20:28:02.840 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:02.840 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250407 values less than (1743955200)) 2025-04-08 20:28:03.079 INFO 3009027 --- [ocp-schedules-11,ee14e44bf4a17802,cea7eda030f3fd32] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:28:03.079 INFO 3009027 --- [ocp-async-10,60df21eced6f0d9b,bf1060f607a8e236] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:28:03.092 WARN 3009027 --- [ocp-async-6,ef91afa06ad4cc9e,c197c9794dab3a7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:03.092 INFO 3009027 --- [ocp-async-8,3bd6a8b80eef8fbd,9785c0859a89a428] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:28:03.177543+08:00, lastProcessTime=2025-04-08T20:27:53.176497+08:00 2025-04-08 20:28:03.331 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:28:03.177543+08:00 2025-04-08 20:28:03.365 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:03.366 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:03.383 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:03.383 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250408 2025-04-08 20:28:03.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:03.384 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250408 values less than (1744041600)) 2025-04-08 20:28:03.569 INFO 3009027 --- [prometheus-minute-schedule-1,a645badc294e2804,a465990d5043b05d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:03.837 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:03.838 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:03.854 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:03.855 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250409 2025-04-08 20:28:03.855 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:03.856 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250409 values less than (1744128000)) 2025-04-08 20:28:04.011 INFO 3009027 --- [ocp-schedules-20,a14e56344e44664e,67279e7586c27a88] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:28:04.019 INFO 3009027 --- [ocp-schedules-17,b4505c383d4fbf9f,9f46b2d24afbfc27] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:28:04.092 INFO 3009027 --- [ocp-schedules-17,b4505c383d4fbf9f,9f46b2d24afbfc27] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:28:04.092 INFO 3009027 --- [ocp-schedules-17,b4505c383d4fbf9f,9f46b2d24afbfc27] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:28:04.380 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:04.381 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:04.398 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:04.399 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250410 2025-04-08 20:28:04.399 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:04.399 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250410 values less than (1744214400)) 2025-04-08 20:28:04.644 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,df6295e209b8884a,9248b4ac3eb00f63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=df6295e209b8884a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 20:28:04.644 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,48c23e77b42def4f,862f3732e53452d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/regions?null, client=10.11.12.102, traceId=48c23e77b42def4f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listRegions(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 20:28:04.644 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,18113457df56499a,fd8ebfeed609ed2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.11.12.102, traceId=18113457df56499a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2025-04-08 20:28:04.658 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,18113457df56499a,fd8ebfeed609ed2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.11.12.102, traceId=18113457df56499a, duration=18 ms] 2025-04-08 20:28:04.659 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,df6295e209b8884a,9248b4ac3eb00f63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=df6295e209b8884a, duration=19 ms] 2025-04-08 20:28:04.662 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,48c23e77b42def4f,862f3732e53452d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/regions client=10.11.12.102, traceId=48c23e77b42def4f, duration=22 ms] 2025-04-08 20:28:04.791 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1d46fce830837064,7eaa7f0e10b16ad0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?page=1&size=10, client=10.11.12.102, traceId=1d46fce830837064, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:28:04.796 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1d46fce830837064,7eaa7f0e10b16ad0] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2025-04-08 20:28:04.815 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1d46fce830837064,7eaa7f0e10b16ad0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.11.12.102, traceId=1d46fce830837064, duration=148 ms] 2025-04-08 20:28:05.092 WARN 3009027 --- [ocp-async-4,47c3067eda7c3dfe,75428a4389d9491c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:05.092 INFO 3009027 --- [ocp-async-10,a1cb9c58134d4b73,d527e4b2c0d9f8e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:05.204 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:05.204 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:05.252 INFO 3009027 --- [ocp-schedules-21,25e94cf695f0b244,0d09223a2465714d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:28:05.253 INFO 3009027 --- [ocp-schedules-21,25e94cf695f0b244,0d09223a2465714d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:28:05.272 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:05.273 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:05.289 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:05.290 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_second, partition=p20250411 2025-04-08 20:28:05.291 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:05.291 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_second` add partition(partition p20250411 values less than (1744300800)) 2025-04-08 20:28:05.589 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,2e134ba21cce4e65,6a1f251f9d246330] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=2e134ba21cce4e65, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:05.592 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,2e134ba21cce4e65,6a1f251f9d246330] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=2e134ba21cce4e65, duration=7 ms] 2025-04-08 20:28:05.619 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:05.620 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_second' and table_type='BASE TABLE' 2025-04-08 20:28:05.635 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_second 2025-04-08 20:28:05.636 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:05.636 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:05.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:05.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 20:28:05.640 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250309 2025-04-08 20:28:05.640 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:05.641 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250309 values less than (1741449600)) 2025-04-08 20:28:06.554 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:06.555 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:06.569 INFO 3009027 --- [prometheus-minute-schedule-1,405e258eb0d82de2,73b67f740b8d608c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:06.572 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:06.573 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250310 2025-04-08 20:28:06.573 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:06.574 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250310 values less than (1741536000)) 2025-04-08 20:28:07.092 WARN 3009027 --- [ocp-async-1,8c7bacd4dc803430,3a93be058b8ffd47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:07.092 INFO 3009027 --- [ocp-async-9,64802796d346969d,d1a9d3c38af786e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:07.138 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:07.139 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:07.155 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:07.156 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250311 2025-04-08 20:28:07.156 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:07.157 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250311 values less than (1741622400)) 2025-04-08 20:28:07.650 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:07.651 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:07.667 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:07.668 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250312 2025-04-08 20:28:07.669 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:07.669 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250312 values less than (1741708800)) 2025-04-08 20:28:08.063 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.064 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:08.080 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:08.080 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250313 2025-04-08 20:28:08.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250313 values less than (1741795200)) 2025-04-08 20:28:08.205 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:08.205 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:08.696 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.697 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:08.714 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:08.715 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250314 2025-04-08 20:28:08.715 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.715 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250314 values less than (1741881600)) 2025-04-08 20:28:08.823 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.824 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:08.841 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:08.842 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250315 2025-04-08 20:28:08.842 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.842 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250315 values less than (1741968000)) 2025-04-08 20:28:08.983 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:08.984 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:09.001 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:09.001 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250316 2025-04-08 20:28:09.002 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.002 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250316 values less than (1742054400)) 2025-04-08 20:28:09.092 WARN 3009027 --- [ocp-async-5,0aee9a6a5f1c953c,c673544e1995bef1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:09.092 INFO 3009027 --- [ocp-async-2,bb1bf63a698cc2a1,a240fd9663664105] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:09.110 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.111 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:09.129 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:09.130 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250317 2025-04-08 20:28:09.130 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.130 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250317 values less than (1742140800)) 2025-04-08 20:28:09.569 INFO 3009027 --- [prometheus-minute-schedule-1,41e4ee0c8f380c21,4c61254a476571e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:09.570 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.570 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:09.587 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:09.587 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250318 2025-04-08 20:28:09.588 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.588 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250318 values less than (1742227200)) 2025-04-08 20:28:09.904 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.905 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:09.921 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:09.921 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250319 2025-04-08 20:28:09.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:09.922 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250319 values less than (1742313600)) 2025-04-08 20:28:10.001 INFO 3009027 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:28:10.001 INFO 3009027 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:28:10.138 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 0, ruleSize = 0. 2025-04-08 20:28:10.154 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.155 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:10.171 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:10.172 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250320 2025-04-08 20:28:10.172 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.173 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250320 values less than (1742400000)) 2025-04-08 20:28:10.479 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.480 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:10.497 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:10.497 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250321 2025-04-08 20:28:10.498 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.498 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250321 values less than (1742486400)) 2025-04-08 20:28:10.769 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:10.786 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:10.786 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250322 2025-04-08 20:28:10.787 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.787 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250322 values less than (1742572800)) 2025-04-08 20:28:10.888 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,e5c3f729009c04d9,e409641e880fdb2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.11.12.102, traceId=e5c3f729009c04d9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,OBCLUSTER,] 2025-04-08 20:28:10.893 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,e5c3f729009c04d9,e409641e880fdb2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=e5c3f729009c04d9, duration=9 ms] 2025-04-08 20:28:10.898 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.899 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:10.903 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,b15dd0dadea673f9,52bcbf265c58b433] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.11.12.102, traceId=b15dd0dadea673f9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:28:10.907 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,b15dd0dadea673f9,52bcbf265c58b433] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.11.12.102, traceId=b15dd0dadea673f9, duration=7 ms] 2025-04-08 20:28:10.915 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:10.916 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250323 2025-04-08 20:28:10.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:10.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250323 values less than (1742659200)) 2025-04-08 20:28:11.092 WARN 3009027 --- [ocp-async-4,273e19f5f78839cd,de9f6d24677b4f90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:11.092 INFO 3009027 --- [ocp-async-8,7b728ae206aa0f4b,56c0aa2d013d3a82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:11.205 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:11.206 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:11.296 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,584c94198b332175,4c03c0b0304d9a3d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A10%2B08%3A00&endTime=2025-04-08T20%3A28%3A10%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=584c94198b332175, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:10+08:00,2025-04-08T20:28:10+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:11.297 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-8,584c94198b332175,4c03c0b0304d9a3d] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:11.298 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,584c94198b332175,4c03c0b0304d9a3d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=584c94198b332175, duration=6 ms] 2025-04-08 20:28:11.301 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,ce694438e2859a32,f8f59860f2d67956] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A10%2B08%3A00&endTime=2025-04-08T20%3A28%3A10%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=ce694438e2859a32, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:10+08:00,2025-04-08T20:28:10+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:11.301 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-7,ce694438e2859a32,f8f59860f2d67956] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = tps_rt, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:11.302 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,ce694438e2859a32,f8f59860f2d67956] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=ce694438e2859a32, duration=5 ms] 2025-04-08 20:28:11.306 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,299d1e01397422a7,932d8da82510aa31] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A10%2B08%3A00&endTime=2025-04-08T20%3A28%3A10%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=299d1e01397422a7, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:10+08:00,2025-04-08T20:28:10+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:11.307 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,299d1e01397422a7,932d8da82510aa31] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_rt, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:11.308 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,299d1e01397422a7,932d8da82510aa31] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=299d1e01397422a7, duration=5 ms] 2025-04-08 20:28:11.320 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,023df44ed128d7b9,fe28aab6256c52c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T19%3A28%3A10%2B08%3A00&endTime=2025-04-08T20%3A28%3A10%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=023df44ed128d7b9, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:10+08:00,2025-04-08T20:28:10+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 20:28:11.320 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,023df44ed128d7b9,fe28aab6256c52c2] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = tps, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 30 2025-04-08 20:28:11.321 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,023df44ed128d7b9,fe28aab6256c52c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=023df44ed128d7b9, duration=5 ms] 2025-04-08 20:28:11.341 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:11.342 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:11.360 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:11.360 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250324 2025-04-08 20:28:11.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:11.361 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250324 values less than (1742745600)) 2025-04-08 20:28:11.623 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:11.624 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:11.640 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:11.641 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250325 2025-04-08 20:28:11.641 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:11.641 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250325 values less than (1742832000)) 2025-04-08 20:28:11.868 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,27be2f358b96ee55,d6df5bd5c6274e6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=27be2f358b96ee55, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:11.871 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,27be2f358b96ee55,d6df5bd5c6274e6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=27be2f358b96ee55, duration=7 ms] 2025-04-08 20:28:12.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:12.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:12.100 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:12.101 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250326 2025-04-08 20:28:12.101 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:12.101 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250326 values less than (1742918400)) 2025-04-08 20:28:12.420 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:12.420 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:12.437 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:12.438 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250327 2025-04-08 20:28:12.438 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:12.438 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250327 values less than (1743004800)) 2025-04-08 20:28:12.550 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,25e9bc7e57768b60,e519797b60dd167e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=25e9bc7e57768b60, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:28:12.550 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,ec951f9bec6034f9,0bf67d94375a46a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=ec951f9bec6034f9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:28:12.550 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,7ced43e745553283,9e5df63fbfe41583] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=7ced43e745553283, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:28:12.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8f59f528da759574,7b0eb7f10fdd9cd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A28%3A12%2B08%3A00&endTime=2025-04-08T20%3A28%3A12%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=8f59f528da759574, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:12+08:00,2025-04-08T20:28:12+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:28:12.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,7ced43e745553283,9e5df63fbfe41583] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=7ced43e745553283, duration=4 ms] 2025-04-08 20:28:12.551 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8f59f528da759574,7b0eb7f10fdd9cd9] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 30 2025-04-08 20:28:12.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,ec951f9bec6034f9,0bf67d94375a46a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=ec951f9bec6034f9, duration=4 ms] 2025-04-08 20:28:12.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,896e9e278f8764ca,edc281435ff1d2c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=896e9e278f8764ca, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:28:12.552 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,25e9bc7e57768b60,e519797b60dd167e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=25e9bc7e57768b60, duration=5 ms] 2025-04-08 20:28:12.554 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8f59f528da759574,7b0eb7f10fdd9cd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=8f59f528da759574, duration=7 ms] 2025-04-08 20:28:12.555 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,896e9e278f8764ca,edc281435ff1d2c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=896e9e278f8764ca, duration=9 ms] 2025-04-08 20:28:12.569 INFO 3009027 --- [prometheus-minute-schedule-1,4a91cf99383b7632,1a56cea7aa025aa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:12.570 INFO 3009027 --- [prometheus-cache-monitor1,94244eed58d6eb9c,23d385929ce2045d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":704,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":386,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:28:12.570 INFO 3009027 --- [prometheus-cache-monitor1,94244eed58d6eb9c,23d385929ce2045d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=49, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, http_server_requests_seconds_count=49, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=33, affectRows=33 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=47, affectRows=47 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=27, affectRows=27 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=43, affectRows=43 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=50, affectRows=50 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=47, affectRows=47 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=56, affectRows=56 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=25, affectRows=25 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=13, affectRows=13 2025-04-08 20:28:12.657 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115220, nodeCount=45, affectRows=45 2025-04-08 20:28:12.659 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=13, validCount=13 2025-04-08 20:28:12.659 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,6e381a5f6d83d107,fee0209185ba819e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=6e381a5f6d83d107, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:28:12.660 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=46, validCount=46 2025-04-08 20:28:12.660 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=47, validCount=47 2025-04-08 20:28:12.660 INFO 3009027 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=33, validCount=33 2025-04-08 20:28:12.660 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=50, validCount=50 2025-04-08 20:28:12.661 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=25, validCount=25 2025-04-08 20:28:12.661 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=27, validCount=27 2025-04-08 20:28:12.661 INFO 3009027 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=56, validCount=56 2025-04-08 20:28:12.661 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=45, validCount=45 2025-04-08 20:28:12.661 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115220, nodeCount=43, validCount=43 2025-04-08 20:28:12.665 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,6e381a5f6d83d107,fee0209185ba819e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=6e381a5f6d83d107, duration=98 ms] 2025-04-08 20:28:12.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:12.917 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:12.932 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:12.933 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250328 2025-04-08 20:28:12.934 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:12.934 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250328 values less than (1743091200)) 2025-04-08 20:28:13.079 INFO 3009027 --- [ocp-schedules-23,1639b1349e595676,8d202981c24a985a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:28:13.079 INFO 3009027 --- [ocp-async-1,ec66f8bd33a34420,0bf82ff7ff035421] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:28:13.092 WARN 3009027 --- [ocp-async-6,77ce705751f88891,405559d902e857fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:13.093 INFO 3009027 --- [ocp-async-10,aea5386f078a1054,a63c69e2ae1d9b18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:28:13.177092+08:00, lastProcessTime=2025-04-08T20:28:03.177543+08:00 2025-04-08 20:28:13.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.218 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:13.234 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:13.234 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250329 2025-04-08 20:28:13.235 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.235 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250329 values less than (1743177600)) 2025-04-08 20:28:13.304 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:28:13.177092+08:00 2025-04-08 20:28:13.469 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,6304a28cb65b0ee9,84058536681b3e64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=6304a28cb65b0ee9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:28:13.469 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,27706dffd4d3039d,2b73b2adcec8c5ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=27706dffd4d3039d, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:28:13.469 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,ba1b7817f8f27ae2,0ddcadb37126f5f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.11.12.102, traceId=ba1b7817f8f27ae2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2025-04-08 20:28:13.469 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,27706dffd4d3039d,2b73b2adcec8c5ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=27706dffd4d3039d, duration=4 ms] 2025-04-08 20:28:13.469 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,6304a28cb65b0ee9,84058536681b3e64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=6304a28cb65b0ee9, duration=4 ms] 2025-04-08 20:28:13.469 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,ba1b7817f8f27ae2,0ddcadb37126f5f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.11.12.102, traceId=ba1b7817f8f27ae2, duration=3 ms] 2025-04-08 20:28:13.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,4b06a24c9d585023,4ae90430af91bdde] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.11.12.102, traceId=4b06a24c9d585023, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2025-04-08 20:28:13.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,bf75e403d50c95db,24ba72ba33019572] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.11.12.102, traceId=bf75e403d50c95db, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2025-04-08 20:28:13.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,4b06a24c9d585023,4ae90430af91bdde] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.11.12.102, traceId=4b06a24c9d585023, duration=3 ms] 2025-04-08 20:28:13.472 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,bf75e403d50c95db,24ba72ba33019572] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.11.12.102, traceId=bf75e403d50c95db, duration=5 ms] 2025-04-08 20:28:13.603 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,197a9593e1b07b72,ab84a2ca13c4e19a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.11.12.102, traceId=197a9593e1b07b72, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2025-04-08 20:28:13.603 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,197a9593e1b07b72,ab84a2ca13c4e19a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.11.12.102, traceId=197a9593e1b07b72, duration=3 ms] 2025-04-08 20:28:13.604 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d206923016c237a8,d94e365c3162d7f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.11.12.102, traceId=d206923016c237a8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-04-08 20:28:13.604 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4c212007631acb9e,844a069517c18055] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP, client=10.11.12.102, traceId=4c212007631acb9e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,OBCLUSTER,] 2025-04-08 20:28:13.605 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c5422d3f8b627cfd,d97ab6541109050e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP, client=10.11.12.102, traceId=c5422d3f8b627cfd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,OBCLUSTER,] 2025-04-08 20:28:13.606 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,c86c180127049687,dd4f50ec0d0c8025] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP, client=10.11.12.102, traceId=c86c180127049687, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,OBCLUSTER,] 2025-04-08 20:28:13.606 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d206923016c237a8,d94e365c3162d7f1] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_cpu_assigned_percentage, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 60 2025-04-08 20:28:13.606 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fa6b43a5f5ffca43,137f54129b0fed67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.11.12.102, traceId=fa6b43a5f5ffca43, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-04-08 20:28:13.606 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d206923016c237a8,d94e365c3162d7f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.11.12.102, traceId=d206923016c237a8, duration=5 ms] 2025-04-08 20:28:13.607 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4c212007631acb9e,844a069517c18055] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=4c212007631acb9e, duration=6 ms] 2025-04-08 20:28:13.608 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fa6b43a5f5ffca43,137f54129b0fed67] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_disk_used_percentage, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 60 2025-04-08 20:28:13.609 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c5422d3f8b627cfd,d97ab6541109050e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=c5422d3f8b627cfd, duration=8 ms] 2025-04-08 20:28:13.610 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,c86c180127049687,dd4f50ec0d0c8025] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=c86c180127049687, duration=9 ms] 2025-04-08 20:28:13.610 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fa6b43a5f5ffca43,137f54129b0fed67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.11.12.102, traceId=fa6b43a5f5ffca43, duration=6 ms] 2025-04-08 20:28:13.621 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.622 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:13.638 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:13.639 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250330 2025-04-08 20:28:13.639 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.639 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250330 values less than (1743264000)) 2025-04-08 20:28:13.651 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,261b14302261cf3a,7bf8f9e72d65fe15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.11.12.102, traceId=261b14302261cf3a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-04-08 20:28:13.651 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,666e394a7cb8870a,9adaca399e128d87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.11.12.102, traceId=666e394a7cb8870a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-04-08 20:28:13.651 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,08338a07e211b905,0954d432d915b20a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.11.12.102, traceId=08338a07e211b905, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-04-08 20:28:13.653 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,08338a07e211b905,0954d432d915b20a] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_memory_assigned_percentage, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 60 2025-04-08 20:28:13.653 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-8,666e394a7cb8870a,9adaca399e128d87] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_memory_percent, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-04-08 20:28:13.653 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,261b14302261cf3a,7bf8f9e72d65fe15] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_cpu_percent, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-04-08 20:28:13.654 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,08338a07e211b905,0954d432d915b20a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.11.12.102, traceId=08338a07e211b905, duration=7 ms] 2025-04-08 20:28:13.654 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,261b14302261cf3a,7bf8f9e72d65fe15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.11.12.102, traceId=261b14302261cf3a, duration=5 ms] 2025-04-08 20:28:13.654 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,666e394a7cb8870a,9adaca399e128d87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.11.12.102, traceId=666e394a7cb8870a, duration=6 ms] 2025-04-08 20:28:13.658 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,09d78cefe725deb5,f1e8c75fbbdcf55e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.11.12.102, traceId=09d78cefe725deb5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-04-08 20:28:13.660 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,740cacef9b17464c,ec25c67d20aa6632] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.11.12.102, traceId=740cacef9b17464c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-04-08 20:28:13.662 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-10,09d78cefe725deb5,f1e8c75fbbdcf55e] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_tenant_disk_usage_v4, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-04-08 20:28:13.663 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,09d78cefe725deb5,f1e8c75fbbdcf55e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.11.12.102, traceId=09d78cefe725deb5, duration=8 ms] 2025-04-08 20:28:13.666 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,740cacef9b17464c,ec25c67d20aa6632] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.11.12.102, traceId=740cacef9b17464c, duration=9 ms] 2025-04-08 20:28:13.805 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.806 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:13.823 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:13.824 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250331 2025-04-08 20:28:13.824 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.825 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250331 values less than (1743350400)) 2025-04-08 20:28:13.984 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:13.985 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:14.001 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:14.002 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250401 2025-04-08 20:28:14.002 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.003 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250401 values less than (1743436800)) 2025-04-08 20:28:14.020 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:27:43+08:00, expireTime=null, isLeader=false} 2025-04-08 20:28:14.020 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:28:14.036 INFO 3009027 --- [ocp-schedules-4,6fafbe9df9bb606e,0a6126d47eeb2878] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:28:14.161 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.162 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:14.178 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:14.179 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250402 2025-04-08 20:28:14.179 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.180 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250402 values less than (1743523200)) 2025-04-08 20:28:14.206 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:14.207 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:14.329 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.330 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:14.346 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:14.347 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250403 2025-04-08 20:28:14.347 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.348 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250403 values less than (1743609600)) 2025-04-08 20:28:14.444 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,19bb98859444f965,fd29999f369bb7c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=19bb98859444f965, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:14.447 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,19bb98859444f965,fd29999f369bb7c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=19bb98859444f965, duration=6 ms] 2025-04-08 20:28:14.548 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.549 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:14.565 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:14.566 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250404 2025-04-08 20:28:14.566 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.567 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250404 values less than (1743696000)) 2025-04-08 20:28:14.752 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.753 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:14.768 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:14.769 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250405 2025-04-08 20:28:14.769 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.770 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250405 values less than (1743782400)) 2025-04-08 20:28:14.947 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.948 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:14.965 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:14.965 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250406 2025-04-08 20:28:14.966 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:14.966 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250406 values less than (1743868800)) 2025-04-08 20:28:15.093 WARN 3009027 --- [ocp-async-10,4d91a969aa27d6dd,697dab7e4edda000] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:15.093 INFO 3009027 --- [ocp-async-1,7c2f870375b9c97e,c10ed548381e29ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:15.310 INFO 3009027 --- [ocp-schedules-25,7e26307f0f980f8b,4edc37e222d5ed64] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:28:15.311 INFO 3009027 --- [ocp-schedules-25,7e26307f0f980f8b,4edc37e222d5ed64] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:28:15.337 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:15.338 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:15.357 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:15.358 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250407 2025-04-08 20:28:15.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:15.358 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250407 values less than (1743955200)) 2025-04-08 20:28:15.557 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:15.558 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:15.569 INFO 3009027 --- [prometheus-minute-schedule-1,9a97694c87da8c22,3ca3f497e5d73e52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:15.574 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:15.575 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250408 2025-04-08 20:28:15.575 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:15.575 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250408 values less than (1744041600)) 2025-04-08 20:28:15.776 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,8f21806b05059df3,6944d38551f68c11] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=8f21806b05059df3, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:28:15.776 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,090bdce891b58a18,c09a45251d768a93] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=090bdce891b58a18, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:28:15.776 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8eabd7c0f62000e1,e4b6f37b4ab0da16] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=8eabd7c0f62000e1, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:28:15.776 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,8f21806b05059df3,6944d38551f68c11] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=8f21806b05059df3, duration=3 ms] 2025-04-08 20:28:15.776 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8eabd7c0f62000e1,e4b6f37b4ab0da16] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=8eabd7c0f62000e1, duration=3 ms] 2025-04-08 20:28:15.776 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c350ce1bc5ecc3b5,43f8d7f674e88e4a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A28%3A15%2B08%3A00&endTime=2025-04-08T20%3A28%3A15%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=c350ce1bc5ecc3b5, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:28:15+08:00,2025-04-08T20:28:15+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:28:15.778 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c350ce1bc5ecc3b5,43f8d7f674e88e4a] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = sql_all_count, labels = , groupBy = app,ob_cluster_name,ob_cluster_id, interval = 30 2025-04-08 20:28:15.778 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,090bdce891b58a18,c09a45251d768a93] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=090bdce891b58a18, duration=5 ms] 2025-04-08 20:28:15.778 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,5bac5ebe37dc91d4,96c502388e15b55c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=5bac5ebe37dc91d4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:28:15.780 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c350ce1bc5ecc3b5,43f8d7f674e88e4a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=c350ce1bc5ecc3b5, duration=7 ms] 2025-04-08 20:28:15.782 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,5bac5ebe37dc91d4,96c502388e15b55c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=5bac5ebe37dc91d4, duration=9 ms] 2025-04-08 20:28:15.829 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,41f2402817ee39b4,464ba12963b6e3bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=41f2402817ee39b4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:28:15.835 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,41f2402817ee39b4,464ba12963b6e3bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=41f2402817ee39b4, duration=41 ms] 2025-04-08 20:28:15.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:15.896 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:15.914 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:15.914 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250409 2025-04-08 20:28:15.915 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:15.915 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250409 values less than (1744128000)) 2025-04-08 20:28:16.334 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.335 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:16.351 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:16.351 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250410 2025-04-08 20:28:16.352 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.352 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250410 values less than (1744214400)) 2025-04-08 20:28:16.614 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.615 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:16.631 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:16.632 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_minute, partition=p20250411 2025-04-08 20:28:16.632 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.632 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_minute` add partition(partition p20250411 values less than (1744300800)) 2025-04-08 20:28:16.751 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,866899e370c4ca05,826a07beb7a586dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=866899e370c4ca05, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:16.754 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,866899e370c4ca05,826a07beb7a586dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=866899e370c4ca05, duration=6 ms] 2025-04-08 20:28:16.894 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.895 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_minute' and table_type='BASE TABLE' 2025-04-08 20:28:16.911 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_minute 2025-04-08 20:28:16.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:16.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.912 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 20:28:16.916 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2022 2025-04-08 20:28:16.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:16.916 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2022 values less than (1672531200)) 2025-04-08 20:28:17.028 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:17.029 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2025-04-08 20:28:17.045 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2025-04-08 20:28:17.046 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2023 2025-04-08 20:28:17.046 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:17.046 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2023 values less than (1704067200)) 2025-04-08 20:28:17.093 WARN 3009027 --- [ocp-async-2,aab14aa417325931,ab64fa9687b8a9f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:17.093 INFO 3009027 --- [ocp-async-3,ffc19cda90db7700,87c19acdeb91a3aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:17.206 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:17.207 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:17.207 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2025-04-08 20:28:17.208 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:17.222 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2025-04-08 20:28:17.223 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2024 2025-04-08 20:28:17.223 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:17.224 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2024 values less than (1735689600)) 2025-04-08 20:28:17.631 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,12545ef4aa61acf6,e59350dd85ed4d0f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=12545ef4aa61acf6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:28:17.631 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,5ed28454d44caf3f,da920c36557ab87d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=5ed28454d44caf3f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:28:17.631 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,5ed28454d44caf3f,da920c36557ab87d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=5ed28454d44caf3f, duration=3 ms] 2025-04-08 20:28:17.632 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,ca47bb01c763effb,8716cfa089395f62] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=ca47bb01c763effb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:28:17.634 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,12545ef4aa61acf6,e59350dd85ed4d0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=12545ef4aa61acf6, duration=6 ms] 2025-04-08 20:28:17.635 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,ca47bb01c763effb,8716cfa089395f62] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=ca47bb01c763effb, duration=7 ms] 2025-04-08 20:28:17.656 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,bb401ba681545068,c7a9b9682b070092] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=bb401ba681545068, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:28:17.743 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,bb401ba681545068,c7a9b9682b070092] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=bb401ba681545068, duration=91 ms] 2025-04-08 20:28:17.802 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,9905269bc9838bd2,21326da988809b6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=9905269bc9838bd2, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:28:17.805 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,9905269bc9838bd2,21326da988809b6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=9905269bc9838bd2, duration=6 ms] 2025-04-08 20:28:17.904 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:28:17.904 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:28:17.904 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:28:18.072 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:18.073 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2025-04-08 20:28:18.089 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2025-04-08 20:28:18.090 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2025 2025-04-08 20:28:18.090 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:18.091 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2025 values less than (1767225600)) 2025-04-08 20:28:18.560 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:18.561 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2025-04-08 20:28:18.569 INFO 3009027 --- [prometheus-minute-schedule-1,a010599151f42444,01982b645b58f2ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:18.577 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2025-04-08 20:28:18.577 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_hour, partition=p2026 2025-04-08 20:28:18.577 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:18.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_hour` add partition(partition p2026 values less than (1798761600)) 2025-04-08 20:28:19.093 WARN 3009027 --- [ocp-async-3,7c289fb4dd229409,685fae8506f5cc92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:19.093 INFO 3009027 --- [ocp-async-5,1eccd7177aa7f6b1,cb23984a4339e442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:19.134 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:19.135 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_hour' and table_type='BASE TABLE' 2025-04-08 20:28:19.151 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_hour 2025-04-08 20:28:19.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:19.152 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:19.153 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:19.153 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 20:28:19.156 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2022 2025-04-08 20:28:19.156 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:19.156 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2022 values less than (1672531200)) 2025-04-08 20:28:19.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,4a4454983c8fe589,1a747f5b2e9ea901] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=4a4454983c8fe589, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:28:19.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,4a4454983c8fe589,1a747f5b2e9ea901] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=4a4454983c8fe589, duration=3 ms] 2025-04-08 20:28:19.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,fff6c1e4d95c9f34,b1e997d449941520] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=fff6c1e4d95c9f34, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:28:19.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,0ed1cba4999bc4d6,a6bd564a0b5a327f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=0ed1cba4999bc4d6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:28:19.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,fff6c1e4d95c9f34,b1e997d449941520] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=fff6c1e4d95c9f34, duration=3 ms] 2025-04-08 20:28:19.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,0ed1cba4999bc4d6,a6bd564a0b5a327f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=0ed1cba4999bc4d6, duration=3 ms] 2025-04-08 20:28:19.761 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:19.762 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2025-04-08 20:28:19.778 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2025-04-08 20:28:19.778 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2023 2025-04-08 20:28:19.779 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:19.779 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2023 values less than (1704067200)) 2025-04-08 20:28:19.786 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:28:19.786 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:28:19.786 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:28:20.109 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:20.110 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2025-04-08 20:28:20.127 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2025-04-08 20:28:20.128 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2024 2025-04-08 20:28:20.128 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:20.128 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2024 values less than (1735689600)) 2025-04-08 20:28:20.208 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:20.208 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:20.818 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:20.819 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2025-04-08 20:28:20.838 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2025-04-08 20:28:20.839 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2025 2025-04-08 20:28:20.839 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:20.839 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2025 values less than (1767225600)) 2025-04-08 20:28:21.093 WARN 3009027 --- [ocp-async-5,079a101c570f1c98,957c4d4bb19e3661] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:21.093 INFO 3009027 --- [ocp-async-9,fbff4385fefd2830,051a7d85fb44705e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:21.216 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:21.217 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2025-04-08 20:28:21.233 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2025-04-08 20:28:21.234 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_daily, partition=p2026 2025-04-08 20:28:21.234 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:21.235 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_daily` add partition(partition p2026 values less than (1798761600)) 2025-04-08 20:28:21.569 INFO 3009027 --- [prometheus-minute-schedule-1,9db9b4c6fc0850fd,fc3cde6e5122a8a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:21.778 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:21.778 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_daily' and table_type='BASE TABLE' 2025-04-08 20:28:21.794 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_daily 2025-04-08 20:28:21.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:21.795 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:21.796 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:21.796 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 20:28:21.799 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20250408 2025-04-08 20:28:21.800 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:21.800 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20250408 values less than (1744041600)) 2025-04-08 20:28:22.343 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:22.344 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2025-04-08 20:28:22.362 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2025-04-08 20:28:22.363 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20250409 2025-04-08 20:28:22.363 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:22.363 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20250409 values less than (1744128000)) 2025-04-08 20:28:22.627 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,d8d8dcf5e055ed2f,513bedf94cc1c754] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=d8d8dcf5e055ed2f, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:28:22.630 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,d8d8dcf5e055ed2f,513bedf94cc1c754] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=d8d8dcf5e055ed2f, duration=6 ms] 2025-04-08 20:28:22.983 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:22.984 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2025-04-08 20:28:23.000 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2025-04-08 20:28:23.001 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20250410 2025-04-08 20:28:23.001 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.001 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20250410 values less than (1744214400)) 2025-04-08 20:28:23.057 INFO 3009027 --- [ocp-schedules-18,07ce00ccbb1c97dd,dc29b56b95ef05d8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:28:23.057+08:00, clusters=[] 2025-04-08 20:28:23.057 INFO 3009027 --- [ocp-schedules-6,3582ee09049cbbdf,6ec8e8fbedb29d06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:28:23.057 INFO 3009027 --- [ocp-schedules-1,7cd1274909affcc3,3c2514c970579ea4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:28:23.057+08:00, clusters=[] 2025-04-08 20:28:23.057 INFO 3009027 --- [ocp-schedules-17,2a7459e0c1791e95,3cdf304b4e03b262] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:28:23.057 INFO 3009027 --- [ocp-schedules-16,5282ad3602139a13,5083ed4cf7ccde9f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:28:23.057 INFO 3009027 --- [ocp-schedules-28,327318b2c6ef2b1e,f2053569ec061d30] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:28:23.058 INFO 3009027 --- [ocp-schedules-5,4cebea97de1fe327,f0387f6ca07cee90] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:28:23.058 INFO 3009027 --- [ocp-schedules-20,de3a8999e6f7faae,071228de82fbecef] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:28:23.060 INFO 3009027 --- [ocp-schedules-23,2de1420d5c3d2a59,6f6fa24b386b696c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:28:23.060 INFO 3009027 --- [ocp-schedules-13,1fa1cf08419051d2,af6658c2227f0ff0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:28:23.060 INFO 3009027 --- [ocp-schedules-23,2de1420d5c3d2a59,6f6fa24b386b696c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:28:23.060 INFO 3009027 --- [ocp-schedules-13,1fa1cf08419051d2,af6658c2227f0ff0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:28:23.060 INFO 3009027 --- [ocp-schedules-21,61d5b385f5563480,e4eec8ee02a10148] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:28:23.061 INFO 3009027 --- [ocp-schedules-21,61d5b385f5563480,e4eec8ee02a10148] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:28:23.065 INFO 3009027 --- [ocp-schedules-28,327318b2c6ef2b1e,f2053569ec061d30] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:28:23.074 INFO 3009027 --- [ocp-schedules-9,33ad89229b27bf83,4e41a9a6cb3ec35c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 20:28:23.074 INFO 3009027 --- [ocp-schedules-9,33ad89229b27bf83,4e41a9a6cb3ec35c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 20:28:23.074 INFO 3009027 --- [ocp-schedules-9,a16e72fcb06b3187,07134b2576d34b06] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":704,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":396,"seriesIdKeySize":396,"secondWriteQueue":0,"minuteWriteCacheSize":385} 2025-04-08 20:28:23.074 INFO 3009027 --- [ocp-schedules-9,a16e72fcb06b3187,07134b2576d34b06] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=54, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, http_server_requests_seconds_count=54, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:28:23.075 INFO 3009027 --- [ocp-schedules-27,c25e4cfab97549b4,cdf93b1a6248aa46] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:28:23.077 INFO 3009027 --- [ocp-schedules-27,c25e4cfab97549b4,cdf93b1a6248aa46] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:28:23.079 INFO 3009027 --- [ocp-schedules-26,c1faa8201e40e76d,bc5a66b89a535916] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:28:23.079 INFO 3009027 --- [ocp-async-9,6ed316193f1ffbf0,8bbd76a5b274b63d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:28:23.080 INFO 3009027 --- [ocp-schedules-8,3667d7871e612c65,becc32cca40b5bd2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:28:23.082 INFO 3009027 --- [ocp-schedules-29,3b3b3d5275897ab1,3d31f449237be307] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:28:23.089 INFO 3009027 --- [ocp-schedules-18,7a34ef85f0b4d528,33b4adb0384e47c6] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:28:23.091 INFO 3009027 --- [ocp-schedules-3,fde04d25b7bab763,d94e9c3f0f35cf66] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:28:23.093 WARN 3009027 --- [ocp-async-3,23da41837fb8e43b,2fdcfe264a61f6ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:23.093 INFO 3009027 --- [ocp-async-4,7b75db39450a0e17,42bf455b8d148c42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:23.106 INFO 3009027 --- [ocp-schedules-23,f03c61b4371f7bb3,8b403a610344f330] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:28:23.113 INFO 3009027 --- [ocp-schedules-13,c71d4f3fa045ca55,cacbfa6ce68e88ce] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:28:23.119 INFO 3009027 --- [ocp-schedules-21,9b0149dac6dde8c9,fc9f922e885f2b1c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:28:23.119 INFO 3009027 --- [ocp-schedules-21,9b0149dac6dde8c9,fc9f922e885f2b1c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:28:23.121 INFO 3009027 --- [ocp-schedules-21,9b0149dac6dde8c9,fc9f922e885f2b1c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:28:23.121 INFO 3009027 --- [ocp-schedules-21,9b0149dac6dde8c9,fc9f922e885f2b1c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:28:23.127 INFO 3009027 --- [ocp-schedules-28,efb296b289bed217,4ac9811a7cd835cc] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:28:23.127 INFO 3009027 --- [ocp-schedules-28,efb296b289bed217,4ac9811a7cd835cc] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:28:23.147 INFO 3009027 --- [ocp-schedules-30,169cadb8d8a03ede,0e876b8c39b25a36] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:28:23.147 INFO 3009027 --- [ocp-schedules-30,169cadb8d8a03ede,0e876b8c39b25a36] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:28:23.148 INFO 3009027 --- [ocp-schedules-9,0af39b3aff9f704c,ecaf0a726f55564f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:28:23.149 INFO 3009027 --- [ocp-schedules-9,0af39b3aff9f704c,ecaf0a726f55564f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:28:23.149 INFO 3009027 --- [ocp-schedules-9,0af39b3aff9f704c,ecaf0a726f55564f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:28:23.156 INFO 3009027 --- [ocp-schedules-16,5651fcd50b7df32e,c05b8602525a5f89] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:28:23.156 INFO 3009027 --- [ocp-schedules-6,3582ee09049cbbdf,6ec8e8fbedb29d06] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:28:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:28:23.177241+08:00, lastProcessTime=2025-04-08T20:28:13.177092+08:00 2025-04-08 20:28:23.191 INFO 3009027 --- [ocp-schedules-27,77af57ef44f2e767,eb71493b45c3cdd9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:28:23.208 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:23.209 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:23.213 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:28:23.213 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:28:23.217 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:28:23.217 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766197260254310400 2025-04-08 20:28:23.222 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=78, maxSyncedRecordId=766197733954809856 2025-04-08 20:28:23.222 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:28:23.222 INFO 3009027 --- [ocp-schedules-26,0aa032fbe42de015,c359168c9ebf3dfe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:28:23.468 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:28:23.177241+08:00 2025-04-08 20:28:23.574 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.575 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2025-04-08 20:28:23.593 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2025-04-08 20:28:23.594 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=metric_data_async, partition=p20250411 2025-04-08 20:28:23.594 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.595 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `metric_data_async` add partition(partition p20250411 values less than (1744300800)) 2025-04-08 20:28:23.696 INFO 3009027 --- [ocp-schedules-7,985329006b206e88,db5e52f9d5fcb97a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:28:23.958 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.959 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='metric_data_async' and table_type='BASE TABLE' 2025-04-08 20:28:23.974 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=metric_data_async 2025-04-08 20:28:23.975 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:23.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.976 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 20:28:23.979 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743436800 2025-04-08 20:28:23.979 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:23.980 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743436800 values less than (1743436800)) 2025-04-08 20:28:24.056 INFO 3009027 --- [ocp-schedules-17,8f99eda999bb522a,d597d67fb97c1adf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:28:24.093 INFO 3009027 --- [ocp-schedules-23,f56e31658b1f3af7,7f814c9749b96538] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:28:24.301 INFO 3009027 --- [ocp-schedules-23,f56e31658b1f3af7,7f814c9749b96538] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:28:24.301 INFO 3009027 --- [ocp-schedules-23,f56e31658b1f3af7,7f814c9749b96538] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:28:24.460 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:24.461 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:24.477 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:24.478 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743523200 2025-04-08 20:28:24.478 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:24.478 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743523200 values less than (1743523200)) 2025-04-08 20:28:24.569 INFO 3009027 --- [prometheus-minute-schedule-1,e7f24532fad2a5f9,a9566de31ff5480d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:25.061 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:25.062 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:25.077 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:25.078 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743609600 2025-04-08 20:28:25.078 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:25.078 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743609600 values less than (1743609600)) 2025-04-08 20:28:25.093 WARN 3009027 --- [ocp-async-1,5e9b034721b848c9,795562b6104903b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:25.094 INFO 3009027 --- [ocp-async-4,fc9e5be1c9fc6df6,9c9c1fe9982230f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:25.403 INFO 3009027 --- [ocp-schedules-10,2c4361a445caa9d5,fbbc93d8f0e2d843] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:28:25.404 INFO 3009027 --- [ocp-schedules-10,2c4361a445caa9d5,fbbc93d8f0e2d843] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:28:25.720 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:25.721 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:25.737 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:25.738 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743696000 2025-04-08 20:28:25.738 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:25.739 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743696000 values less than (1743696000)) 2025-04-08 20:28:26.024 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:26.025 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:26.041 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:26.042 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743782400 2025-04-08 20:28:26.042 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:26.043 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743782400 values less than (1743782400)) 2025-04-08 20:28:26.209 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:26.210 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:26.499 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:26.500 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:26.516 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:26.517 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743868800 2025-04-08 20:28:26.517 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:26.517 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743868800 values less than (1743868800)) 2025-04-08 20:28:27.094 WARN 3009027 --- [ocp-async-3,2787a60a4ac0037c,66c76c1e52a6d47f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:27.094 INFO 3009027 --- [ocp-async-10,461491f454152b64,08c8fe195aaebd96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:27.139 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:28:27.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:27.145 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:27.161 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:27.161 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1743955200 2025-04-08 20:28:27.162 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:27.162 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1743955200 values less than (1743955200)) 2025-04-08 20:28:27.569 INFO 3009027 --- [prometheus-minute-schedule-1,79ce73948137e009,c140fc6e0dd0e624] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:27.853 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:27.854 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:27.871 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:27.872 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1744041600 2025-04-08 20:28:27.872 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:27.873 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1744041600 values less than (1744041600)) 2025-04-08 20:28:28.279 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:28.280 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:28.296 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:28.297 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1744128000 2025-04-08 20:28:28.297 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:28.298 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1744128000 values less than (1744128000)) 2025-04-08 20:28:28.637 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:28.638 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:28.654 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:28.655 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1744214400 2025-04-08 20:28:28.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:28.655 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1744214400 values less than (1744214400)) 2025-04-08 20:28:29.094 WARN 3009027 --- [ocp-async-10,33e172fa5e06e8b4,ba46652b56935e4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:29.094 INFO 3009027 --- [ocp-async-7,58c7b894343febc3,104a0be992f63bca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:29.117 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:29.118 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:29.134 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:29.135 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=sql_aggregate_data, partition=p1744300800 2025-04-08 20:28:29.136 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:29.136 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `sql_aggregate_data` add partition(partition p1744300800 values less than (1744300800)) 2025-04-08 20:28:29.210 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:29.210 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:29.555 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:29.556 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='sql_aggregate_data' and table_type='BASE TABLE' 2025-04-08 20:28:29.577 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=sql_aggregate_data 2025-04-08 20:28:29.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:29.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:29.579 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:29.579 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 20:28:29.582 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2022 2025-04-08 20:28:29.582 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:29.583 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2022 values less than (1672531200)) 2025-04-08 20:28:30.122 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:30.123 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2025-04-08 20:28:30.143 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2025-04-08 20:28:30.144 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2023 2025-04-08 20:28:30.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:30.144 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2023 values less than (1704067200)) 2025-04-08 20:28:30.569 INFO 3009027 --- [prometheus-minute-schedule-1,129905e90d467e2d,f957453ff675cafe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:30.850 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:30.851 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2025-04-08 20:28:30.868 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2025-04-08 20:28:30.869 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2024 2025-04-08 20:28:30.869 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:30.869 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2024 values less than (1735689600)) 2025-04-08 20:28:31.094 WARN 3009027 --- [ocp-async-1,8dff6ec1d2fd1a47,44547ca7c6cc9c68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:31.094 INFO 3009027 --- [ocp-async-9,3230ecf5371c6c71,91759ab5343b0c2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:31.526 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:31.527 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2025-04-08 20:28:31.543 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2025-04-08 20:28:31.544 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2025 2025-04-08 20:28:31.544 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:31.544 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2025 values less than (1767225600)) 2025-04-08 20:28:32.210 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:32.211 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:32.296 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:32.297 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2025-04-08 20:28:32.313 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2025-04-08 20:28:32.314 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_hist_partition_stats_compressed, partition=p2026 2025-04-08 20:28:32.314 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:32.315 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_hist_partition_stats_compressed` add partition(partition p2026 values less than (1798761600)) 2025-04-08 20:28:33.080 INFO 3009027 --- [ocp-schedules-23,0072a049bfb35038,d386ea0b06736910] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:28:33.080 INFO 3009027 --- [ocp-async-9,e66e36ca2a737b59,d27e69028efca1f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:28:33.094 WARN 3009027 --- [ocp-async-6,1d1deedde39bb2bd,198a263d0bf531e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:33.094 INFO 3009027 --- [ocp-async-8,81bda4cb7627e623,e7602f12312e9e80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:33.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:28:33.177787+08:00, lastProcessTime=2025-04-08T20:28:23.177241+08:00 2025-04-08 20:28:33.372 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:28:33.177787+08:00 2025-04-08 20:28:33.554 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1239, sql=alter table `ob_hist_partition_stats_compressed` add partition(partition p2026 values less than (1798761600)) 2025-04-08 20:28:33.555 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:33.556 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_hist_partition_stats_compressed' and table_type='BASE TABLE' 2025-04-08 20:28:33.569 INFO 3009027 --- [prometheus-minute-schedule-1,24b3a604c9ac14d5,e18245d9b661844b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:33.572 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_hist_partition_stats_compressed 2025-04-08 20:28:33.573 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:33.573 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:28:33.573 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:33.574 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 20:28:33.578 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1743523200 2025-04-08 20:28:33.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:33.578 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1743523200 values less than (1743523200)) 2025-04-08 20:28:34.089 INFO 3009027 --- [ocp-schedules-13,019815a3eca186fc,a1d9569c568bcd67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:28:34.282 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:34.283 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:34.299 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:34.299 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1743609600 2025-04-08 20:28:34.300 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:34.300 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1743609600 values less than (1743609600)) 2025-04-08 20:28:35.080 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:35.081 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:35.094 WARN 3009027 --- [ocp-async-9,f87a5db58db52867,2d1725acd573fc2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:35.094 INFO 3009027 --- [ocp-async-7,08887beb5961c5bb,61f08c75ef1769e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:35.098 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:35.098 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1743696000 2025-04-08 20:28:35.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:35.099 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1743696000 values less than (1743696000)) 2025-04-08 20:28:35.211 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:35.212 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:35.478 INFO 3009027 --- [ocp-schedules-5,fdb1c89420df9cfc,9bae02d695d02966] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:28:35.480 INFO 3009027 --- [ocp-schedules-5,fdb1c89420df9cfc,9bae02d695d02966] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:28:35.954 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOverPreCheck, client 10.11.12.102, traceId 28ccdd46fcc32478] 2025-04-08 20:28:35.987 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOverPreCheck?null, client=10.11.12.102, traceId=28ccdd46fcc32478, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(TakeOverClusterPreCheckParam), args=TakeOverClusterPreCheckParam(connectionMode=direct, clusterName=null, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=192.168.1.61, port=2881),] 2025-04-08 20:28:36.000 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=192.168.1.61, port=2881, database=oceanbase) 2025-04-08 20:28:36.000 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 20:28:36.006 INFO 3009027 --- [obsdk-init-thread-3,28ccdd46fcc32478,cec91a0bc82c4f4b] com.alibaba.druid.pool.DruidDataSource : {dataSource-5,ob-connector-datasource_direct_sys_root_192.168.1.61_2881} inited 2025-04-08 20:28:36.006 INFO 3009027 --- [obsdk-init-thread-3,28ccdd46fcc32478,cec91a0bc82c4f4b] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 20:28:36.006 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 20:28:36.007 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.008 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.016 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.016 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.020 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.020 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.021 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.021 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.022 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.023 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.027 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.028 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 20:28:36.034 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.035 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.035 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.036 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 20:28:36.037 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.038 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:36.041 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.041 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.042 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.042 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 20:28:36.047 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.048 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.049 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.050 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.054 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 20:28:36.055 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:36.056 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1743782400 2025-04-08 20:28:36.056 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.057 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1743782400 values less than (1743782400)) 2025-04-08 20:28:36.059 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.059 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.060 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.060 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.061 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.061 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 20:28:36.068 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.068 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.074 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 20:28:36.100 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.100 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.104 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.107 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 20:28:36.113 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.113 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.115 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.115 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.121 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:28:36.121 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.125 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:28:36.148 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.148 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.152 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.153 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:28:36.176 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.176 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.177 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.177 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.178 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.178 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.179 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.179 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.180 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.180 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:28:36.200 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.200 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.205 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.205 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:28:36.211 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.211 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.212 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.213 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:28:36.245 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.245 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.246 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.246 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.247 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.248 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.249 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.249 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.250 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.250 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:28:36.267 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.268 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.269 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.269 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:28:36.272 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.273 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.274 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.274 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:28:36.291 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.292 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.293 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.293 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.294 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.294 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.295 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.295 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.296 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.296 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:28:36.312 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.313 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.314 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.314 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:28:36.333 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.334 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.335 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.335 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.336 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.336 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.337 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.337 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 20:28:36.387 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.388 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.389 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.389 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.391 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.392 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:28:36.393 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.393 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:28:36.394 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.394 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:28:36.401 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.401 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.402 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.402 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.403 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.403 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.404 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.404 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:28:36.405 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.405 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:28:36.432 ERROR 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 20:28:36.437 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,28ccdd46fcc32478,50f068d044dceac0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.11.12.102, traceId=28ccdd46fcc32478, duration=485 ms] 2025-04-08 20:28:36.540 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.541 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:36.558 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:36.559 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1743868800 2025-04-08 20:28:36.559 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:36.560 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1743868800 values less than (1743868800)) 2025-04-08 20:28:36.569 INFO 3009027 --- [prometheus-minute-schedule-1,daac522c37670f61,4be8d319d9d7f4ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:36.621 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c2c7fb8c8e3ad2c7,6a9283f4e065ed3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=c2c7fb8c8e3ad2c7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 20:28:36.622 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,722b33bc5098ae3e,84508322fb0ba18e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=722b33bc5098ae3e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 20:28:36.623 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c2c7fb8c8e3ad2c7,6a9283f4e065ed3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=c2c7fb8c8e3ad2c7, duration=5 ms] 2025-04-08 20:28:36.677 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,722b33bc5098ae3e,84508322fb0ba18e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=722b33bc5098ae3e, duration=59 ms] 2025-04-08 20:28:37.094 WARN 3009027 --- [ocp-async-3,0fdd8834ab277b1d,83fed3b22bf8b9e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:37.095 INFO 3009027 --- [ocp-async-7,b0bc78b5b0eacf09,0804952cdeac5b69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:37.431 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:37.432 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:37.448 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:37.449 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1743955200 2025-04-08 20:28:37.449 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:37.450 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1743955200 values less than (1743955200)) 2025-04-08 20:28:38.188 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:38.189 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:38.206 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:38.207 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1744041600 2025-04-08 20:28:38.207 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:38.208 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1744041600 values less than (1744041600)) 2025-04-08 20:28:38.212 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:38.213 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:39.002 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:39.003 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:39.019 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:39.020 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1744128000 2025-04-08 20:28:39.020 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:39.020 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1744128000 values less than (1744128000)) 2025-04-08 20:28:39.095 WARN 3009027 --- [ocp-async-7,135185df6290cb40,80391113e21f3e38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:39.095 INFO 3009027 --- [ocp-async-5,1e476f0acf6d726c,a182385610eb0f67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:39.569 INFO 3009027 --- [prometheus-minute-schedule-1,7aa71291e36227f0,f2ff73f836f4eb23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:39.841 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:39.842 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:39.858 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:39.858 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1744214400 2025-04-08 20:28:39.859 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:39.859 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1744214400 values less than (1744214400)) 2025-04-08 20:28:40.001 INFO 3009027 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:28:40.002 INFO 3009027 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:28:40.831 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:40.831 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:40.847 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:40.848 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167014] Creating partition, scope=Monitordb, table=ob_tenant_mini_compaction_info, partition=p1744300800 2025-04-08 20:28:40.848 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:40.848 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: alter table `ob_tenant_mini_compaction_info` add partition(partition p1744300800 values less than (1744300800)) 2025-04-08 20:28:41.095 WARN 3009027 --- [ocp-async-5,b34d271f3c9dd949,78ae070a8b54c067] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:41.095 INFO 3009027 --- [ocp-async-9,6934e315de25ef76,d6926214de2841c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:41.131 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:41.135 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:28:41.136 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:41.136 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:28:41.137 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:28:41.138 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 20:28:41.213 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:41.214 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:41.701 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:28:41.702 INFO 3009027 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select table_schema databaseName, table_name tableName, create_time gmtCreate from information_schema.tables where table_schema='monitor_database' and table_name='ob_tenant_mini_compaction_info' and table_type='BASE TABLE' 2025-04-08 20:28:41.718 INFO 3009027 --- [part-initial-worker,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm can not be resoled because alarm target is not found, alarmType=partition_create_failed, target=alarm_template_id=0:datasource=Monitordb:table_name=ob_tenant_mini_compaction_info 2025-04-08 20:28:41.719 INFO 3009027 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 20:28:42.569 INFO 3009027 --- [prometheus-minute-schedule-1,28ce9ac353b03769,42b51df090ad3cb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:43.080 INFO 3009027 --- [ocp-schedules-13,fdb8f89ac1f428ff,7adcede5bddccd39] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:28:43.080 INFO 3009027 --- [ocp-async-8,1c7f4838dc11ce6d,57e1ade5ddd12d3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:28:43.095 WARN 3009027 --- [ocp-async-2,52f211e4915c3207,4e765e04636550fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:43.095 INFO 3009027 --- [ocp-async-9,574eb70dab2d84bd,ca3451edadc6d79d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:28:43.175769+08:00, lastProcessTime=2025-04-08T20:28:33.177787+08:00 2025-04-08 20:28:43.422 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:28:43.175769+08:00 2025-04-08 20:28:44.107 INFO 3009027 --- [ocp-schedules-11,dd6fff1302b325d9,9cc874b421f0ed4c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:28:44.214 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:44.214 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:44.303 INFO 3009027 --- [ocp-schedules-12,1308324e9336f1a7,e8cc7278b29e607a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:28:44.321 INFO 3009027 --- [ocp-schedules-12,1308324e9336f1a7,e8cc7278b29e607a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:28:44.321 INFO 3009027 --- [ocp-schedules-12,1308324e9336f1a7,e8cc7278b29e607a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:28:45.095 WARN 3009027 --- [ocp-async-5,7bc37a324e2b2a26,a468f2cbdeccfffb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:45.095 INFO 3009027 --- [ocp-async-10,b2323531dd550ee8,bb5844507776bda6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:45.569 INFO 3009027 --- [prometheus-minute-schedule-1,20879cfe0cc3ff5f,12d6ee7bd7e660bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:45.615 INFO 3009027 --- [ocp-schedules-14,942b175543da3f54,1be3f6e4c48d40de] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:28:45.616 INFO 3009027 --- [ocp-schedules-14,942b175543da3f54,1be3f6e4c48d40de] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:28:47.095 WARN 3009027 --- [ocp-async-9,3dc83040a4fedec8,f0d42cb70d668853] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:47.095 INFO 3009027 --- [ocp-async-10,c4b0d867b760d55c,0ab31f8e921f8385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:47.215 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:47.215 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:48.569 INFO 3009027 --- [prometheus-minute-schedule-1,7446be6459b1c0c0,9586fa471f4f08fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:49.095 WARN 3009027 --- [ocp-async-6,d8644d45818e27bb,6a6c10e2c938b13d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:49.095 INFO 3009027 --- [ocp-async-10,d452439c7ca14bec,54894721021da328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:50.215 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:50.216 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:51.096 WARN 3009027 --- [ocp-async-7,3e7ad9984fc393c8,6d3994374965b1e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:51.096 INFO 3009027 --- [ocp-async-8,f611770dfded48e4,80d77b52273346dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:51.197 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,89ab8bfa83cc76a9,6d432757e14201ae] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/compute/hostTypes, client 10.11.12.102, traceId 89ab8bfa83cc76a9] 2025-04-08 20:28:51.203 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,89ab8bfa83cc76a9,6d432757e14201ae] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=89ab8bfa83cc76a9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ComputeController.createHostType(CreateHostTypeRequest), args=CreateHostTypeRequest(name=GSCCloud, description=null),] 2025-04-08 20:28:51.248 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,89ab8bfa83cc76a9,6d432757e14201ae] c.o.o.c.h.service.HostTypeServiceImpl : Created hostType: 1 GSCCloud 2025-04-08 20:28:51.248 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,89ab8bfa83cc76a9,6d432757e14201ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/compute/hostTypes client=10.11.12.102, traceId=89ab8bfa83cc76a9, duration=53 ms] 2025-04-08 20:28:51.249 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,89ab8bfa83cc76a9,6d432757e14201ae] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=CREATE_HOST_TYPE, eventType=COMPUTE_HOST_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ComputeController.createHostType, resourceType=COMPUTE_HOST, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=GSCCloud, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"hostTypeRequest":{"name":"GSCCloud"}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/compute/hostTypes, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T20:28:51.203+08:00, endTime=2025-04-08T20:28:51.248+08:00, role=null, aliUid=null) 2025-04-08 20:28:51.325 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,fdda96df140614c1,6fb51ab9b36810ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=fdda96df140614c1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 20:28:51.326 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,fdda96df140614c1,6fb51ab9b36810ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=fdda96df140614c1, duration=5 ms] 2025-04-08 20:28:51.569 INFO 3009027 --- [prometheus-minute-schedule-1,255070620e1515d9,afe8f492047dbeb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:53.057 INFO 3009027 --- [ocp-schedules-10,66837d1057ea3b63,005fdd0369896f4b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:28:53.057 INFO 3009027 --- [ocp-schedules-20,a24462794d0cdb4c,9021ef123674760c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:28:53.058 INFO 3009027 --- [ocp-schedules-20,807e55048d5362ca,baa61269b47a3fd1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:28:53.058 INFO 3009027 --- [ocp-schedules-5,849c1aa3e5cfc2ef,5dfa84d564513cdb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:28:53.060 INFO 3009027 --- [ocp-schedules-23,3c3259ccc0697310,e0ed8b5f40ed92b7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:28:53.060 INFO 3009027 --- [ocp-schedules-14,1ab8ff19c15a518d,6ce2e38845c4ba22] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:28:53.060 INFO 3009027 --- [ocp-schedules-23,3c3259ccc0697310,e0ed8b5f40ed92b7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:28:53.060 INFO 3009027 --- [ocp-schedules-14,1ab8ff19c15a518d,6ce2e38845c4ba22] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:28:53.060 INFO 3009027 --- [ocp-schedules-29,6402b0c83db46602,5e2c58f1079b8f0c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:28:53.061 INFO 3009027 --- [ocp-schedules-29,6402b0c83db46602,5e2c58f1079b8f0c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:28:53.080 INFO 3009027 --- [ocp-schedules-6,dc64d95979639840,e206c4e9380aa22a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:28:53.080 INFO 3009027 --- [ocp-async-9,4c6e8d734b8ddcbe,d71bdad69454b9ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:28:53.096 WARN 3009027 --- [ocp-async-1,f0e50e39bc70e3ce,36ea19e36c8ea346] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:53.096 INFO 3009027 --- [ocp-async-8,4cf27271d6735629,0a8c20fd90b8586f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:53.100 INFO 3009027 --- [ocp-schedules-13,519d05056cebe943,99315459eba718da] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:28:53.114 INFO 3009027 --- [ocp-schedules-19,40c0ee37689e6596,d154e24525ee15f1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:28:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:28:53.177269+08:00, lastProcessTime=2025-04-08T20:28:43.175769+08:00 2025-04-08 20:28:53.216 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:53.216 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:53.378 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:28:53.177269+08:00 2025-04-08 20:28:54.144 INFO 3009027 --- [ocp-schedules-2,3234b75bf14eea5d,37b9031dfad3d5ab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:28:54.431 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,18c914b8ecec8de9,fdacf5d62f4b7e43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=18c914b8ecec8de9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 20:28:54.431 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,91fc75fa00f7dc2c,84773aad6a3d9cbb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.11.12.102, traceId=91fc75fa00f7dc2c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 20:28:54.433 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-8,91fc75fa00f7dc2c,84773aad6a3d9cbb] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2025-04-08 20:28:54.434 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,18c914b8ecec8de9,fdacf5d62f4b7e43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=18c914b8ecec8de9, duration=7 ms] 2025-04-08 20:28:54.435 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,91fc75fa00f7dc2c,84773aad6a3d9cbb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.11.12.102, traceId=91fc75fa00f7dc2c, duration=8 ms] 2025-04-08 20:28:54.569 INFO 3009027 --- [prometheus-minute-schedule-1,061e518ddba8608b,af33a1b934a4aec9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:55.096 WARN 3009027 --- [ocp-async-7,94054cbef1158977,60481fa750f1382f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:55.096 INFO 3009027 --- [ocp-async-5,d3680b03c50a49e4,a19ab9ec00b1ec8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:55.649 INFO 3009027 --- [ocp-schedules-27,7b0a4f44eb022e2c,9c57a9a258343def] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:28:55.650 INFO 3009027 --- [ocp-schedules-27,7b0a4f44eb022e2c,9c57a9a258343def] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:28:56.217 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:56.217 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:28:57.096 WARN 3009027 --- [ocp-async-8,3be8aef50ecb6201,8fdf9b3b57c25871] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:57.096 INFO 3009027 --- [ocp-async-4,bee76b46c75fd26e,232e2a77c0ba8a8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:57.569 INFO 3009027 --- [prometheus-minute-schedule-1,524d125e3df10904,f6455b69919d2fb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115340, interval=60 2025-04-08 20:28:59.096 WARN 3009027 --- [ocp-async-3,8c44eb6b2c1def0e,856054a2f35c1c7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:28:59.096 INFO 3009027 --- [ocp-async-4,3ad59050cae3dd97,b3c7c3751edc8a4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:28:59.218 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:28:59.218 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:00.569 INFO 3009027 --- [prometheus-minute-schedule-1,4e52eb61d57d365a,526c843a88c72b77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:01.096 WARN 3009027 --- [ocp-async-7,c079dc85f9a5c5bf,4d0d0652b758afda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:01.096 INFO 3009027 --- [ocp-async-4,0567506a842dd887,efe99574587145c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:02.219 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:02.219 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:03.080 INFO 3009027 --- [ocp-schedules-10,2d624c56ae2ebb63,d9bf60d60056a619] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:29:03.081 INFO 3009027 --- [ocp-async-9,d48edc525776eae3,9b7f1a7b6a8ffb7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:29:03.097 WARN 3009027 --- [ocp-async-5,cbb8bd0ea2eb8f27,383c657273427b04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:03.097 INFO 3009027 --- [ocp-async-1,26ab7044e6533274,e6b65c89c13d1348] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:29:03.176224+08:00, lastProcessTime=2025-04-08T20:28:53.177269+08:00 2025-04-08 20:29:03.286 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:29:03.176224+08:00 2025-04-08 20:29:03.569 INFO 3009027 --- [prometheus-minute-schedule-1,9d3d6f39d9fe03ac,01d3aef46b3f9ed6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:04.161 INFO 3009027 --- [ocp-schedules-5,1efe14bf5a629f55,7f5cdc0c9080283f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:29:04.323 INFO 3009027 --- [ocp-schedules-10,1a6b32238ceb7e99,04a35953d743c3f1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:29:04.369 INFO 3009027 --- [ocp-schedules-10,1a6b32238ceb7e99,04a35953d743c3f1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:29:04.369 INFO 3009027 --- [ocp-schedules-10,1a6b32238ceb7e99,04a35953d743c3f1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:29:05.097 WARN 3009027 --- [ocp-async-6,b9d18696b1675692,14c320523bffb6cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:05.097 INFO 3009027 --- [ocp-async-3,a3a3b0b0510cfe2d,8bdd5ef9f3f78344] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:05.219 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:05.220 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:05.714 INFO 3009027 --- [ocp-schedules-24,ad397547abf19662,b53b725375cfca13] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:29:05.715 INFO 3009027 --- [ocp-schedules-24,ad397547abf19662,b53b725375cfca13] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:29:06.569 INFO 3009027 --- [prometheus-minute-schedule-1,acf31bde51fb3ea3,7f0d4f1595991a71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:07.097 WARN 3009027 --- [ocp-async-10,8443b9d7cc67add7,ce66dcf5d1bddc19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:07.097 INFO 3009027 --- [ocp-async-3,130c3227a3b302b7,03fd9018726c8b04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:08.220 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:08.221 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:09.097 WARN 3009027 --- [ocp-async-8,48172c1904ce0d20,ea616f350572c774] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:09.097 INFO 3009027 --- [ocp-async-5,168c4acbfcc9f422,c0c7962ce9ffab04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:09.569 INFO 3009027 --- [prometheus-minute-schedule-1,987a73afd1e5c30e,4d68631db2c73344] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:10.001 INFO 3009027 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:29:10.002 INFO 3009027 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:29:10.138 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 0, ruleSize = 0. 2025-04-08 20:29:11.097 WARN 3009027 --- [ocp-async-7,0ae84c441547d459,f2916c5048365003] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:11.097 INFO 3009027 --- [ocp-async-5,113d9c4ac6fbd279,b315a74d4eac295f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:11.221 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:11.222 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:12.569 INFO 3009027 --- [prometheus-minute-schedule-1,b5ed4213dc862ef5,355ea6c07ac6f6cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:12.570 INFO 3009027 --- [prometheus-cache-monitor1,4525ee4642705e74,3d2f08b5966af7a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1108,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":404,"secondWriteQueue":0,"minuteWriteCacheSize":385} 2025-04-08 20:29:12.570 INFO 3009027 --- [prometheus-cache-monitor1,4525ee4642705e74,3d2f08b5966af7a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=57, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, http_server_requests_seconds_count=57, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:29:12.718 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115280, nodeCount=61, affectRows=61 2025-04-08 20:29:12.718 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115280, nodeCount=73, affectRows=73 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115280, nodeCount=64, affectRows=64 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115280, nodeCount=69, affectRows=69 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115280, nodeCount=69, affectRows=69 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115280, nodeCount=68, affectRows=68 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115280, nodeCount=61, validCount=61 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115280, nodeCount=68, validCount=68 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115280, nodeCount=73, validCount=73 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115280, nodeCount=64, validCount=64 2025-04-08 20:29:12.719 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115280, nodeCount=68, validCount=68 2025-04-08 20:29:12.720 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115280, nodeCount=69, validCount=69 2025-04-08 20:29:12.776 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:25:12.415, end=2025-04-08T20:27:12.719} 2025-04-08 20:29:12.781 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:29:12.781 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:29:13.081 INFO 3009027 --- [ocp-schedules-3,1c3677be35c655f0,69af99ad05f08c92] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:29:13.081 INFO 3009027 --- [ocp-async-2,73aebaafc21bf7f0,086ecc031d94ae7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:29:13.097 WARN 3009027 --- [ocp-async-4,3874a90c601f6eb8,a9e35075e53d5ee3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:13.097 INFO 3009027 --- [ocp-async-3,2600ca442d80011a,c2059e88b367f5cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:13.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:29:13.175513+08:00, lastProcessTime=2025-04-08T20:29:03.176224+08:00 2025-04-08 20:29:13.370 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:29:13.175513+08:00 2025-04-08 20:29:14.184 INFO 3009027 --- [ocp-schedules-18,80c018f4b6f394c3,e8b46fbc47c7be6e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:29:14.222 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:14.223 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:15.097 WARN 3009027 --- [ocp-async-2,b171baf44b4c87e3,25b043a8472564be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:15.097 INFO 3009027 --- [ocp-async-1,1a74b5c9e5353e2c,0359d61525212407] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:15.569 INFO 3009027 --- [prometheus-minute-schedule-1,9577e0ff4799498a,f3ac59d9f9d1957a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:15.734 INFO 3009027 --- [ocp-schedules-10,f3cfaf8b271dd8e7,10f0fa9116f40109] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:29:15.735 INFO 3009027 --- [ocp-schedules-10,f3cfaf8b271dd8e7,10f0fa9116f40109] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:29:17.098 WARN 3009027 --- [ocp-async-10,24e2cecebb20f117,b086c76ec84db882] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:17.098 INFO 3009027 --- [ocp-async-4,096ff1b25ebf0be5,e032f164a11d5987] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:17.223 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:17.223 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:18.569 INFO 3009027 --- [prometheus-minute-schedule-1,9ee3173d8e859b22,0f6b590bd4ceead2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:19.098 WARN 3009027 --- [ocp-async-1,3c4855b6e1152218,937f39344b9f87f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:19.098 INFO 3009027 --- [ocp-async-5,f3a0135c73ba8de8,6ef24f5f4ed5f6f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:20.223 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:20.224 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:21.098 WARN 3009027 --- [ocp-async-9,2b5c980ad2d22e41,7a371ec369383241] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:21.098 INFO 3009027 --- [ocp-async-5,9a92dfda1c1fc4ba,e7ac096e55127bcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:21.569 INFO 3009027 --- [prometheus-minute-schedule-1,49d50d45f3d10e66,008995781cdfc758] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:23.057 INFO 3009027 --- [ocp-schedules-21,ff488570b26faa98,877c49b17f93587d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:29:23.057 INFO 3009027 --- [ocp-schedules-8,07a50b6d9109f0b9,9ac981aaa547c8df] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:29:23.057 INFO 3009027 --- [ocp-schedules-29,da6411eb58724a29,c0bc5ae54068f7cd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:29:23.058 INFO 3009027 --- [ocp-schedules-22,2dcdf682a7c8adb0,e2c8b2c975ae6a60] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:29:23.058 INFO 3009027 --- [ocp-schedules-24,7a31ac5f04abf288,b24f7611ba68165c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:29:23.058 INFO 3009027 --- [ocp-schedules-26,52b0603570f2f84b,285b177548afe86c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:29:23.059 INFO 3009027 --- [ocp-schedules-1,183e883ab09f5900,df7828c36532cb1d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:29:23.059+08:00, clusters=[] 2025-04-08 20:29:23.059 INFO 3009027 --- [ocp-schedules-17,714187a2f0381ebe,533adf3d52c4b4a4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:29:23.059+08:00, clusters=[] 2025-04-08 20:29:23.060 INFO 3009027 --- [ocp-schedules-6,24846010d88e9044,b5e203e7c5cbc3a1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:29:23.060 INFO 3009027 --- [ocp-schedules-6,24846010d88e9044,b5e203e7c5cbc3a1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:29:23.060 INFO 3009027 --- [ocp-schedules-7,7fef67850167280f,682f753e36b33619] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:29:23.060 INFO 3009027 --- [ocp-schedules-9,53b1a29fe959932b,275e934e216fa834] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:29:23.060 INFO 3009027 --- [ocp-schedules-7,7fef67850167280f,682f753e36b33619] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:29:23.060 INFO 3009027 --- [ocp-schedules-9,53b1a29fe959932b,275e934e216fa834] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:29:23.064 INFO 3009027 --- [ocp-schedules-22,2dcdf682a7c8adb0,e2c8b2c975ae6a60] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:29:23.075 INFO 3009027 --- [pool-overview-container-8,1137c4502c42184f,56b7260787b95fde] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:23.076 INFO 3009027 --- [pool-overview-container-1,aa148c914277da74,ab7abe874ace063b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:23.077 INFO 3009027 --- [ocp-schedules-23,ad2401300c47fbe0,9372c73c3386cd58] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:29:23.078 INFO 3009027 --- [ocp-schedules-30,1662f0ef8a34faed,0545233af859251d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 20:29:23.078 INFO 3009027 --- [ocp-schedules-30,1662f0ef8a34faed,0545233af859251d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 20:29:23.080 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:29:23.080 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:29:23.081 INFO 3009027 --- [ocp-schedules-28,8ec0f90e9ee9a4fd,19ca6758457914ab] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:29:23.081 INFO 3009027 --- [ocp-schedules-15,968df1a5092ff0f2,71ba64d4b3995daf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:29:23.081 INFO 3009027 --- [ocp-async-5,246cba8861b15934,e8b2ae47cf71f561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:29:23.082 INFO 3009027 --- [ocp-schedules-18,e75510b31d83524f,de30f8f7738d3f52] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:29:23.084 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:29:23.084 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:29:23.084 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:29:23.085 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:29:23.085 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:29:23.085 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:29:23.087 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:29:23.087 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:29:23.087 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:29:23.088 INFO 3009027 --- [ocp-schedules-25,63fbb86dd571c9e4,95716ea2da702298] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:29:23.091 INFO 3009027 --- [ocp-schedules-17,5877f79fd5801286,bf16e623de1cae38] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:29:23.098 INFO 3009027 --- [ocp-async-8,dd31e07941a0ebf4,c657d4037410fe65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:23.098 WARN 3009027 --- [ocp-async-1,a1f803e8458a9603,4a00daec01bee5ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:23.100 INFO 3009027 --- [ocp-schedules-21,ff488570b26faa98,877c49b17f93587d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:29:23.104 INFO 3009027 --- [ocp-schedules-29,7d6b6339e6d2b0fa,399d9cedb95a5827] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:29:23.105 INFO 3009027 --- [ocp-schedules-10,4f84783143bae4eb,9801e0374dafa1ed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:29:23.107 INFO 3009027 --- [ocp-schedules-11,7862fa367ab9a994,e6aae585d4a433f4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:29:23.115 INFO 3009027 --- [ocp-schedules-27,8439c3c31b960215,5c9ebfaba2a1dc49] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:29:23.116 INFO 3009027 --- [ocp-schedules-14,bcc796acc2e27154,184ec6b4e08ade16] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:29:23.116 INFO 3009027 --- [ocp-schedules-14,bcc796acc2e27154,184ec6b4e08ade16] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:29:23.117 INFO 3009027 --- [ocp-schedules-14,bcc796acc2e27154,184ec6b4e08ade16] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:29:23.121 INFO 3009027 --- [ocp-schedules-5,39025aa39b63bc6f,2f8531c68e54cea4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:29:23.121 INFO 3009027 --- [ocp-schedules-5,39025aa39b63bc6f,2f8531c68e54cea4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:29:23.122 INFO 3009027 --- [ocp-schedules-5,39025aa39b63bc6f,2f8531c68e54cea4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:29:23.122 INFO 3009027 --- [ocp-schedules-5,39025aa39b63bc6f,2f8531c68e54cea4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:29:23.129 INFO 3009027 --- [ocp-schedules-4,15b337ee77d7e374,e1339a0e2188192e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:29:23.129 INFO 3009027 --- [ocp-schedules-4,15b337ee77d7e374,e1339a0e2188192e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:29:23.149 INFO 3009027 --- [ocp-schedules-30,18ce08aba70cfdc4,36eea76c9d0a129e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:29:23.149 INFO 3009027 --- [ocp-schedules-23,3a0aa73fe91b6902,61fcceebc3cbe15d] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:29:23.149 INFO 3009027 --- [ocp-schedules-23,3a0aa73fe91b6902,61fcceebc3cbe15d] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:29:23.150 INFO 3009027 --- [ocp-schedules-30,18ce08aba70cfdc4,36eea76c9d0a129e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:29:23.150 INFO 3009027 --- [ocp-schedules-30,18ce08aba70cfdc4,36eea76c9d0a129e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:29:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:29:23.176873+08:00, lastProcessTime=2025-04-08T20:29:13.175513+08:00 2025-04-08 20:29:23.192 INFO 3009027 --- [ocp-schedules-3,8bd23a24ad7eac5b,5e11ee9c11e96b1a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:29:23.224 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:23.225 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:23.225 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:29:23.225 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:29:23.226 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:29:23.226 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766197733954809856 2025-04-08 20:29:23.226 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=8, maxSyncedRecordId=766197921368895488 2025-04-08 20:29:23.226 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:29:23.226 INFO 3009027 --- [ocp-schedules-20,bb22f2e56c8bd665,112ee03cc4a1dc80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:29:23.315 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:29:23.176873+08:00 2025-04-08 20:29:23.712 INFO 3009027 --- [ocp-schedules-26,a7bfa5982d4ece09,ea53532bb1dbd730] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:29:24.202 INFO 3009027 --- [ocp-schedules-11,2f85c56c67a1dd66,ea989a9b13fcf065] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:29:24.371 INFO 3009027 --- [ocp-schedules-14,406de0b6a544c407,2b97b7dd894592d1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:29:24.390 INFO 3009027 --- [ocp-schedules-14,406de0b6a544c407,2b97b7dd894592d1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:29:24.390 INFO 3009027 --- [ocp-schedules-14,406de0b6a544c407,2b97b7dd894592d1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:29:24.569 INFO 3009027 --- [prometheus-minute-schedule-1,bba2542611a628f4,d3101409b85cdfb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:25.098 WARN 3009027 --- [ocp-async-5,ffe3b700e17b97cb,2df1d6ff93889914] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:25.098 INFO 3009027 --- [ocp-async-9,af6c45cf4b5d308f,59863368ba4b551e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:25.745 INFO 3009027 --- [ocp-schedules-11,a84046082f833d70,d64fe3af5b7c6b9f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:29:25.746 INFO 3009027 --- [ocp-schedules-11,a84046082f833d70,d64fe3af5b7c6b9f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:29:26.225 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:26.225 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:27.098 WARN 3009027 --- [ocp-async-1,0bfa78bb109a8208,8cf96c45503ca892] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:27.098 INFO 3009027 --- [ocp-async-10,ecb8f589ee158a3c,c3c80b832e7fbca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:27.139 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:29:27.569 INFO 3009027 --- [prometheus-minute-schedule-1,9cb06e73b759abdd,361f4547e6024fd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:29.098 WARN 3009027 --- [ocp-async-10,5aeb1c48dac1c6f0,a825108288214ca9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:29.099 INFO 3009027 --- [ocp-async-6,5ca4950b1b0f71c4,1ecd2c7288ec6ce7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:29.225 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:29.226 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:29.761 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials, client 10.11.12.102, traceId aa50f40bf5b88dcf] 2025-04-08 20:29:29.775 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials?null, client=10.11.12.102, traceId=aa50f40bf5b88dcf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.createCredential(CreateCredentialParam), args=CreateCredentialParam(targetType=HOST, validate=true, name=rootpingzheng, description=, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=root, passphrase=xxx hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null),] 2025-04-08 20:29:29.780 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=HOST, validate=true, name=rootpingzheng, description=, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=root, passphrase=Xxf7713123#$^%, hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null) 2025-04-08 20:29:29.780 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.o.s.c.CredentialServiceImpl : Validating credential:ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=root, passphrase=Xxf7713123#$^%, hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null, batchSshValidateProperty=null) 2025-04-08 20:29:30.048 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=HOST, validate=true, name=rootpingzheng, description=, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=root, passphrase=Xxf7713123#$^%, hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null) 2025-04-08 20:29:30.050 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials client=10.11.12.102, traceId=aa50f40bf5b88dcf, duration=291 ms] 2025-04-08 20:29:30.051 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,aa50f40bf5b88dcf,df76e3a0ffaf5ed8] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=CREATE_CREDENTIAL, eventType=CREDENTIAL_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.CredentialController.createCredential, resourceType=CREDENTIAL, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=rootpingzheng, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"param":{"description":"","name":"rootpingzheng","obJdbcCredentialProperty":{},"sshCredentialProperty":{"allowSudo":true,"hostIdList":[],"passphrase":"******","type":"PASSWORD","username":"root"},"targetType":"HOST","validate":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/profiles/me/credentials, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T20:29:29.775+08:00, endTime=2025-04-08T20:29:30.050+08:00, role=null, aliUid=null) 2025-04-08 20:29:30.167 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,e094112f89168ddc,cb964024885a93bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=e094112f89168ddc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 20:29:30.172 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,e094112f89168ddc,cb964024885a93bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=e094112f89168ddc, duration=9 ms] 2025-04-08 20:29:30.569 INFO 3009027 --- [prometheus-minute-schedule-1,3f32907723b66ff5,b43d8b234d73bd2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:31.099 WARN 3009027 --- [ocp-async-3,6f009e022d327400,b19ca5d7228f4057] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:31.099 INFO 3009027 --- [ocp-async-5,da74e9907b18a0d7,917d931905fda26e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:32.226 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:32.227 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:33.081 INFO 3009027 --- [ocp-schedules-10,55b1ddb575b0ec37,0a3795d6d024b852] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:29:33.081 INFO 3009027 --- [ocp-async-1,825d8de0bdf193eb,29b85fb24e96396e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:29:33.099 WARN 3009027 --- [ocp-async-9,ab96c4ca495a835d,f0b208549726a271] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:33.099 INFO 3009027 --- [ocp-async-10,d6046c59ea280b30,2e919fdc1ee94116] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:29:33.176556+08:00, lastProcessTime=2025-04-08T20:29:23.176873+08:00 2025-04-08 20:29:33.329 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:29:33.176556+08:00 2025-04-08 20:29:33.569 INFO 3009027 --- [prometheus-minute-schedule-1,9ce88e89087b2125,bbea11c21eb8df46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:34.236 INFO 3009027 --- [ocp-schedules-30,f0f5fc323c8888f7,33cfd48cfb72fc3d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:29:35.099 WARN 3009027 --- [ocp-async-8,3ee88aca3c19d8cb,3296751965f704ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:35.099 INFO 3009027 --- [ocp-async-3,aefe7520eff2d4c8,61a81bce8ee294d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:35.227 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:35.227 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:35.833 INFO 3009027 --- [ocp-schedules-19,c04997fce5dba237,2555c51c07bb3479] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:29:35.834 INFO 3009027 --- [ocp-schedules-19,c04997fce5dba237,2555c51c07bb3479] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:29:36.569 INFO 3009027 --- [prometheus-minute-schedule-1,3e25bd27348931af,b3df23b584eec724] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:37.099 WARN 3009027 --- [ocp-async-7,f4b2708cf1846b6a,e6a4b546b82ee689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:37.099 INFO 3009027 --- [ocp-async-3,6518b39dc9d33d5f,fa35419a2613dc0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:38.228 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:38.228 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:39.099 WARN 3009027 --- [ocp-async-4,4c1d63e89fc9ef27,48c6db949cdda17e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:39.099 INFO 3009027 --- [ocp-async-9,3b1c10eed57d206b,1b82df2fe995df78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:39.569 INFO 3009027 --- [prometheus-minute-schedule-1,d44f161906ab6034,bdcb9ee8c655854b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:40.001 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:29:40.001 INFO 3009027 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:29:41.100 WARN 3009027 --- [ocp-async-6,a5d81aa3d06589e6,08292cf85499d32f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:41.100 INFO 3009027 --- [ocp-async-8,e570727d4995edc2,110141ef264d0069] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:41.229 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:41.229 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:42.569 INFO 3009027 --- [prometheus-minute-schedule-1,0e1455bd07e4ad39,b3796dcb656004e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:43.081 INFO 3009027 --- [ocp-schedules-6,5ccd2876a354bbd0,f1f22a052f5beafa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:29:43.081 INFO 3009027 --- [ocp-async-10,5a1756095b79c323,0aab08071295d1e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:29:43.100 WARN 3009027 --- [ocp-async-3,f059961f5bfc0c9d,d4b9ca413e017066] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:43.100 INFO 3009027 --- [ocp-async-7,03cd0b393951c7b0,fec5895d61d84fe1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:29:43.177052+08:00, lastProcessTime=2025-04-08T20:29:33.176556+08:00 2025-04-08 20:29:43.290 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:29:43.177052+08:00 2025-04-08 20:29:44.146 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:29:13+08:00, expireTime=null, isLeader=false} 2025-04-08 20:29:44.146 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 20:29:44.229 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:44.230 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:44.256 INFO 3009027 --- [ocp-schedules-26,e560ff64ca9837b2,ae392dde05b35ab7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:29:44.391 INFO 3009027 --- [ocp-schedules-9,fc2bf2074c0c1b9f,22e6ab7af49dcf90] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:29:44.413 INFO 3009027 --- [ocp-schedules-9,fc2bf2074c0c1b9f,22e6ab7af49dcf90] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:29:44.413 INFO 3009027 --- [ocp-schedules-9,fc2bf2074c0c1b9f,22e6ab7af49dcf90] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:29:45.100 WARN 3009027 --- [ocp-async-2,626fae8c3eabefb6,ade02a0f175d55a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:45.100 INFO 3009027 --- [ocp-async-9,5a306dbac91c9163,337c806da1671ad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:45.569 INFO 3009027 --- [prometheus-minute-schedule-1,1bee83c4a8f5a423,99283cefaa242568] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:45.845 INFO 3009027 --- [ocp-schedules-4,861eef999a1af0fd,fa2090451e539b2a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:29:45.846 INFO 3009027 --- [ocp-schedules-4,861eef999a1af0fd,fa2090451e539b2a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:29:46.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,7fe248841737312c,c1b7a2e460dda4bb] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId 7fe248841737312c] 2025-04-08 20:29:46.476 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,7fe248841737312c,c1b7a2e460dda4bb] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=7fe248841737312c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 20:29:46.486 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,7fe248841737312c,c1b7a2e460dda4bb] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)] 2025-04-08 20:29:46.492 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] c.o.o.e.internal.connector.Connectors : init global connector holder. 2025-04-08 20:29:46.492 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] c.o.o.e.i.connector.ConnectorHolder : Init connector holder, maxIdleSecs=3600 2025-04-08 20:29:46.799 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 20:29:46.941 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 20:29:46.948 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 20:29:47.100 WARN 3009027 --- [ocp-async-1,f050ef132036cbba,b3b61ef777e9eea3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:47.100 INFO 3009027 --- [ocp-async-6,bcab6ed86e50647e,25cb323f18fa439b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:47.136 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.61:22 success 2025-04-08 20:29:47.174 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@192.168.1.61 2025-04-08 20:29:47.175 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.61, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:1 2025-04-08 20:29:47.200 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:47.224 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:47.230 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:47.231 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:47.257 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:47.275 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:47.275 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:47.276 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:47.293 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:47.293 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.61 2025-04-08 20:29:47.321 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:29:47.321 INFO 3009027 --- [pool-21-thread-1,7fe248841737312c,fa0b3580bfb77600] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:29:47.323 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,7fe248841737312c,c1b7a2e460dda4bb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=7fe248841737312c, duration=855 ms] 2025-04-08 20:29:47.427 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOver, client 10.11.12.102, traceId 73ac1c39155401d0] 2025-04-08 20:29:47.439 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOver?null, client=10.11.12.102, traceId=73ac1c39155401d0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster(TakeOverClusterParam), args=TakeOverClusterParam(switchConfigUrl=true, hostInfo=TakeOverClusterParam.HostInfo(kind=DEDICATED_PHYSICAL_MACHINE, publishPorts=null, hostTypeId=1, sshPort=22, credentialId=1)),] 2025-04-08 20:29:47.449 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-192.168.1.61#2881, method=takeOverCluster 2025-04-08 20:29:47.453 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.454 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.454 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.456 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.456 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.457 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.457 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.457 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.458 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 20:29:47.463 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.464 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.464 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.465 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 20:29:47.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.471 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.471 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 20:29:47.476 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.477 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.477 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.478 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.479 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.479 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 20:29:47.479 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.480 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.480 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.480 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.481 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.481 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 20:29:47.486 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.487 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.487 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.488 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 20:29:47.488 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.489 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.489 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.492 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 20:29:47.493 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.493 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.494 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.494 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.495 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:29:47.495 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.498 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:29:47.502 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.503 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.504 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.504 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:29:47.523 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.524 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.524 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.525 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.526 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.526 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.527 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.527 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.528 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.528 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:29:47.548 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.548 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.549 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.549 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:29:47.550 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.552 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:29:47.568 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.568 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.569 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.569 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.570 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.570 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.571 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.571 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.572 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.572 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:29:47.588 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.589 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.590 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.591 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:29:47.591 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.592 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.592 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.593 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:29:47.608 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.608 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.609 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.609 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.609 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.610 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.611 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.611 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.612 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.613 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:29:47.629 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.629 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.630 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.631 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:29:47.637 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.637 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.638 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.638 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.638 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.639 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.639 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.640 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 20:29:47.651 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.651 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.652 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.652 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.653 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.653 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:29:47.654 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.654 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:29:47.655 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.655 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:29:47.659 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.660 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.660 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.660 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.661 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.661 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.662 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.662 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:47.662 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:47.663 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:29:47.670 ERROR 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 20:29:47.719 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.host.service.RegionServiceImpl : Created region: 2 deault_region 2025-04-08 20:29:47.745 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.host.service.IdcServiceImpl : Created idc: 1 default_idc 2025-04-08 20:29:47.940 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1, resourceId=1 2025-04-08 20:29:47.958 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.o.i.c.ClusterGroupServiceImpl : create cluster group, name=ocpdb, description=ocpdb, create by cluster service, affectedRows=1 2025-04-08 20:29:47.979 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1, resourceId=1 2025-04-08 20:29:48.099 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:29:48.224 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:29:48.224 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:29:48.226 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.CredentialServiceImpl : Credential clusterName=ocpdb|tenantName=sys|username=root is already exists 2025-04-08 20:29:48.226 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.CredentialServiceImpl : Delete credential:clusterName=ocpdb|tenantName=sys|username=root 2025-04-08 20:29:48.234 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:29:48.235 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.235 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:29:48.236 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.236 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:29:48.237 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.237 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:29:48.241 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.241 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:48.242 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.243 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:48.243 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.244 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:48.245 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.245 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:29:48.246 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:29:48.246 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:29:48.251 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.o.i.c.ClusterOperationServiceImpl : meta cluster detected. 2025-04-08 20:29:48.295 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE APP GROUP, property:UpdateAppGroupProperty(serialNumbers=[null], appGroup=null) 2025-04-08 20:29:48.353 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:29:48.360 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Take over primary OB cluster, nodeCount=39, taskCount=39 2025-04-08 20:29:48.360 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, createTime=null, startTime=2025-04-08T20:29:48.360+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Create random agent credential, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=16, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait cluster accessible, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.WaitClusterAccessibleTask, seriesId=18, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=28, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Uninstall legacy ocp agent, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp 2025-04-08 20:29:48.439 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=7} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.459 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=16} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.461 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=18} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.462 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=29} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.472 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=2} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.473 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=26} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.474 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=32} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.479 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=34} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.480 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=23} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.481 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=30} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.482 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=3} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.483 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=20} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:29:48.486 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=2, taskName=Take over primary OB cluster, argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}} 2025-04-08 20:29:48.561 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOver client=10.11.12.102, traceId=73ac1c39155401d0, duration=1136 ms] 2025-04-08 20:29:48.561 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,73ac1c39155401d0,eaaca74aa4b3fe0d] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=2, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T20:29:47.439+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 20:29:48.569 INFO 3009027 --- [prometheus-minute-schedule-1,9b1d039773a086e6,14066004225a4453] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:49.101 WARN 3009027 --- [ocp-async-6,295f3cb45792f8eb,5c36bd764718a3b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:49.101 INFO 3009027 --- [ocp-async-4,d64d8ec65ef5c67c,548ac23427183245] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:49.178 INFO 3009027 --- [subtask-coordinator1,1308668168866431,8f3ff5ffa53b73e1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 22, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:49.246 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 20:29:49.247 INFO 3009027 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 20:29:49.249 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 20:29:49.258 INFO 3009027 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db success. 2025-04-08 20:29:49.273 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:29:49.268 INFO 3009027 --- [manual-subtask-executor1,73ac1c39155401d0,c85d8af8b98c1c21] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:49.275 INFO 3009027 --- [manual-subtask-executor1,73ac1c39155401d0,c85d8af8b98c1c21] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=22, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, service_name=ocpdb:1744114127, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, latest_execution_start_time=2025-04-08T20:29:49.173+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=22}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:49.285 INFO 3009027 --- [manual-subtask-executor1,73ac1c39155401d0,c85d8af8b98c1c21] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 22, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:49.295 INFO 3009027 --- [manual-subtask-executor1,73ac1c39155401d0,c85d8af8b98c1c21] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[20] 2025-04-08 20:29:50.039 INFO 3009027 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1 2025-04-08 20:29:50.223 INFO 3009027 --- [subtask-coordinator1,1308668168866431,759ede341b887ceb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 20, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:50.231 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:50.231 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:50.250 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:50.259 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=20, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:50.221+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=20}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:50.357 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:50.368 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,c06349ef396d96d6,46ece883304807af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=c06349ef396d96d6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:29:50.368 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,af5acc809c379251,663031e33e26dff4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=af5acc809c379251, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:29:50.368 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,c06349ef396d96d6,46ece883304807af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=c06349ef396d96d6, duration=36 ms] 2025-04-08 20:29:50.371 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,af5acc809c379251,663031e33e26dff4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=af5acc809c379251, duration=39 ms] 2025-04-08 20:29:50.377 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,4d72414a7f631173,90ed800bdf5c704f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=4d72414a7f631173, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:29:50.378 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,4d72414a7f631173,90ed800bdf5c704f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=4d72414a7f631173, duration=4 ms] 2025-04-08 20:29:50.383 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.388 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,cf881932dd9db541,e6322ee636860a59] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=cf881932dd9db541, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:29:50.387 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.61 2025-04-08 20:29:50.433 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,eaf2cbefd3816c6d,c4b6dba5c06a7a28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=eaf2cbefd3816c6d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:50.417 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:50.453 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.470 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,eaf2cbefd3816c6d,c4b6dba5c06a7a28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=eaf2cbefd3816c6d, duration=41 ms] 2025-04-08 20:29:50.493 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.61 2025-04-08 20:29:50.505 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1e412baeb13b75fc,32f9ee436f1d25c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=1e412baeb13b75fc, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:29:50.514 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.528 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,cf881932dd9db541,e6322ee636860a59] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=cf881932dd9db541, duration=145 ms] 2025-04-08 20:29:50.532 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1e412baeb13b75fc,32f9ee436f1d25c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=1e412baeb13b75fc, duration=29 ms] 2025-04-08 20:29:50.522 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.593 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:29:50.593 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:29:50.593 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:29:50.540 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 20:29:50.633 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.61, port=22, loopCount=1, minSuccessCount=1 2025-04-08 20:29:50.637 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:50.657 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.745 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.61 2025-04-08 20:29:50.805 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.817 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:50.840 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[204] 2025-04-08 20:29:50.854 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=204, maxAcceptableDelayMillis=3000 2025-04-08 20:29:50.858 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.61], timeoutMillis=20000 2025-04-08 20:29:50.888 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744115390 0 0 , error= 2025-04-08 20:29:50.948 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 20:29:50.959 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 20, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:50.961 INFO 3009027 --- [manual-subtask-executor2,73ac1c39155401d0,7a44ecb6fc90de38] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[21] 2025-04-08 20:29:51.083 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,728b15b4e06ead36,8f3f8136253bdee2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/20/logSegments?taskInstanceId=2&subtaskInstanceId=20, client=10.11.12.102, traceId=728b15b4e06ead36, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,20,] 2025-04-08 20:29:51.101 INFO 3009027 --- [ocp-async-4,3a1eedcb28ea3758,781e0a68c12c4873] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:51.101 WARN 3009027 --- [ocp-async-7,8c7f4129b2da66bc,43c5b5ed74c3fcce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:51.102 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,728b15b4e06ead36,8f3f8136253bdee2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/20/logSegments client=10.11.12.102, traceId=728b15b4e06ead36, duration=23 ms] 2025-04-08 20:29:51.248 INFO 3009027 --- [subtask-coordinator1,1308668168866431,a347d360d67687fe] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:51.263 INFO 3009027 --- [manual-subtask-executor3,73ac1c39155401d0,608910f184233492] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:51.265 INFO 3009027 --- [manual-subtask-executor3,73ac1c39155401d0,608910f184233492] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=21, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:51.247+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=21}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:51.277 INFO 3009027 --- [manual-subtask-executor3,73ac1c39155401d0,608910f184233492] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:51.279 INFO 3009027 --- [manual-subtask-executor3,73ac1c39155401d0,608910f184233492] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[18] 2025-04-08 20:29:51.461 INFO 3009027 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=192.168.1.61:8080, consumerId=3, resources=[ShardingSlice(consumerId=3, type=OB_CLUSTER, resourceIds=[1]), ShardingSlice(consumerId=3, type=HOST_MONITOR, resourceIds=[1])] 2025-04-08 20:29:51.569 INFO 3009027 --- [prometheus-minute-schedule-1,bd37c9a32dacca03,a83986e55466c238] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:51.931 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a1c32bcaf2019a4f,b187559d5e0da72a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=a1c32bcaf2019a4f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:51.947 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a1c32bcaf2019a4f,b187559d5e0da72a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=a1c32bcaf2019a4f, duration=20 ms] 2025-04-08 20:29:52.178 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,52cf32f1242b07a6,7ab1567912380fa7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/20/logSegments?taskInstanceId=2&subtaskInstanceId=20, client=10.11.12.102, traceId=52cf32f1242b07a6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,20,] 2025-04-08 20:29:52.184 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,52cf32f1242b07a6,7ab1567912380fa7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/20/logSegments client=10.11.12.102, traceId=52cf32f1242b07a6, duration=9 ms] 2025-04-08 20:29:52.262 INFO 3009027 --- [subtask-coordinator1,1308668168866431,85b1ce7671c3d509] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 18, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:52.376 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1dada7b484947c3b,c017f5b452f8fb5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=1dada7b484947c3b, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:29:52.376 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1dada7b484947c3b,c017f5b452f8fb5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=1dada7b484947c3b, duration=2 ms] 2025-04-08 20:29:52.376 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,f2c02ff2bc21cd57,daf7db65e3f7f75a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=f2c02ff2bc21cd57, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:29:52.376 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4bd88cb4b9a0f010,4a906fce20128407] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=4bd88cb4b9a0f010, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:29:52.376 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,f2c02ff2bc21cd57,daf7db65e3f7f75a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=f2c02ff2bc21cd57, duration=2 ms] 2025-04-08 20:29:52.376 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4bd88cb4b9a0f010,4a906fce20128407] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=4bd88cb4b9a0f010, duration=2 ms] 2025-04-08 20:29:52.351 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:52.397 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=18, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:52.260+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=18}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:52.484 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:29:52.484 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:29:52.484 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:29:52.482 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 20:29:52.494 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:52.516 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:52.556 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:52.577 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:52.623 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.61 2025-04-08 20:29:52.669 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:52.672 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:52.736 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 20:29:52.753 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 18, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:52.755 INFO 3009027 --- [manual-subtask-executor4,73ac1c39155401d0,5de15249baf31f03] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[27] 2025-04-08 20:29:53.057 INFO 3009027 --- [ocp-schedules-21,db4d1fa9ef044fc7,05a44a75d029f65d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:29:53.057 INFO 3009027 --- [ocp-schedules-2,d6ceebdd2ed22758,02022196f9c1a7a8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:29:53.058 INFO 3009027 --- [ocp-schedules-1,ee3bf03376934d21,6266a7411494b265] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:29:53.059 INFO 3009027 --- [ocp-schedules-2,d6ceebdd2ed22758,02022196f9c1a7a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:29:53.059 INFO 3009027 --- [ocp-schedules-1,ee3bf03376934d21,6266a7411494b265] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:29:53.059 INFO 3009027 --- [ocp-schedules-16,e75098d5526e3e55,e47468a48261fec0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:29:53.060 INFO 3009027 --- [ocp-schedules-16,e75098d5526e3e55,e47468a48261fec0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:29:53.060 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:29:53.060 INFO 3009027 --- [ocp-schedules-26,f518a7781b42840d,9b18a0dc11f97e49] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:29:53.061 INFO 3009027 --- [ocp-schedules-26,f518a7781b42840d,9b18a0dc11f97e49] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:29:53.061 INFO 3009027 --- [ocp-schedules-29,ed5bdefd6a4b084d,67521bfef2921f07] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:29:53.061 INFO 3009027 --- [ocp-schedules-11,d276e7ea6640f8f7,2f6600e4e9aa5a89] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:29:53.061 INFO 3009027 --- [ocp-schedules-11,d276e7ea6640f8f7,2f6600e4e9aa5a89] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:29:53.062 INFO 3009027 --- [ocp-schedules-29,ed5bdefd6a4b084d,67521bfef2921f07] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:29:53.064 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_status_check', labels={app=OB, check_item=cluster connection, failed_reason=-, ob_cluster_id=1744114127, ob_cluster_name=ocpdb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744115393 : 2025-04-08 12:29:53 2025-04-08 20:29:53.081 INFO 3009027 --- [ocp-schedules-14,e5f7b2990fc545b9,97686e0d5ae42666] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:29:53.081 INFO 3009027 --- [ocp-async-9,99c11c09b8011309,3e7878752c545e1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:29:53.088 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:29:53.088 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:29:53.088 INFO 3009027 --- [Sync all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_sync', labels={app=OB, failed_reason=-, ob_cluster_id=1744114127, ob_cluster_name=ocpdb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744115393 : 2025-04-08 12:29:53 2025-04-08 20:29:53.101 WARN 3009027 --- [ocp-async-3,a9be7b5c26360ffd,39e361294fdea7c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:53.101 INFO 3009027 --- [ocp-async-1,6f27ef041fd93f8f,243e002de8806ddf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:53.116 INFO 3009027 --- [ocp-schedules-10,a92262877c045fcd,b09c25e405c12655] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:29:53.155 INFO 3009027 --- [ocp-schedules-25,d3f96ffd820de797,cc15e133f52af46e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:29:53.170 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,f30d29c0442f0765,d8a3e599af8682f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=f30d29c0442f0765, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:29:53.176104+08:00, lastProcessTime=2025-04-08T20:29:43.177052+08:00 2025-04-08 20:29:53.182 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,f30d29c0442f0765,d8a3e599af8682f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=f30d29c0442f0765, duration=16 ms] 2025-04-08 20:29:53.231 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:53.232 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:53.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 20:29:53.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:29:53.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 20:29:53.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:29:53.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 20:29:53.331 INFO 3009027 --- [subtask-coordinator1,1308668168866431,983bc0e6c1adc361] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 27, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:53.351 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:29:53.176104+08:00 2025-04-08 20:29:53.364 INFO 3009027 --- [manual-subtask-executor5,73ac1c39155401d0,f5cddcc63466f535] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:53.367 INFO 3009027 --- [manual-subtask-executor5,73ac1c39155401d0,f5cddcc63466f535] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=27, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:53.329+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=27}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:53.373 INFO 3009027 --- [manual-subtask-executor5,73ac1c39155401d0,f5cddcc63466f535] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 27, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:53.374 INFO 3009027 --- [manual-subtask-executor5,73ac1c39155401d0,f5cddcc63466f535] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[23] 2025-04-08 20:29:53.961 INFO 3009027 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-target, name=192.168.1.61:8080, consumerId=1, resources=[ShardingSlice(consumerId=1, type=OBCluster, resourceIds=[1]), ShardingSlice(consumerId=1, type=Host, resourceIds=[1])] 2025-04-08 20:29:54.284 INFO 3009027 --- [ocp-schedules-14,c6bdfbd0d2671c5b,2dc50973c06e6294] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:29:54.360 INFO 3009027 --- [subtask-coordinator1,1308668168866431,b44ba4e983a92333] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 23, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:54.368 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:54.414 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=23, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:54.358+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=23}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:54.419 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 20:29:54.431 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:54.449 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:54.453 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,393937503d1d651a,a4278375d172e197] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=393937503d1d651a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:54.457 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:54.465 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,393937503d1d651a,a4278375d172e197] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=393937503d1d651a, duration=16 ms] 2025-04-08 20:29:54.476 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:54.549 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:54.569 INFO 3009027 --- [prometheus-minute-schedule-1,1bb82e0e01520a33,4dc5286ae4437c53] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:54.585 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:54.619 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:54.625 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:54.653 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:54.693 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61 2025-04-08 20:29:55.101 WARN 3009027 --- [ocp-async-5,f2528e9040b87933,0088ca8012a62dbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:55.101 INFO 3009027 --- [ocp-async-1,ede8658da6048f28,ceba99cfddd39a5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:55.061 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:55.149 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:55.155 WARN 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 20:29:55.158 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 20:29:55.161 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:55.181 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:55.263 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61 2025-04-08 20:29:55.351 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,b0efb0c86eb9f934,8b0177d597b6109e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=b0efb0c86eb9f934, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:29:55.353 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,b0efb0c86eb9f934,8b0177d597b6109e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=b0efb0c86eb9f934, duration=4 ms] 2025-04-08 20:29:55.711 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,26afb097a5534ed3,e03b58b2acf7503e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=26afb097a5534ed3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:55.724 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,26afb097a5534ed3,e03b58b2acf7503e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=26afb097a5534ed3, duration=17 ms] 2025-04-08 20:29:55.723 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:55.795 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:55.841 WARN 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 20:29:55.844 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 20:29:55.859 INFO 3009027 --- [ocp-schedules-27,f021f7951e00dec4,9a84befb79a9144e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:29:55.860 INFO 3009027 --- [ocp-schedules-27,f021f7951e00dec4,9a84befb79a9144e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:29:55.848 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 20:29:55.950 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 23, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:55.951 INFO 3009027 --- [manual-subtask-executor6,73ac1c39155401d0,ebcdf2a6eb795ef3] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[8] 2025-04-08 20:29:56.232 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:56.233 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:56.369 INFO 3009027 --- [subtask-coordinator1,1308668168866431,dd497a5169717d13] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 8, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:56.382 INFO 3009027 --- [manual-subtask-executor7,73ac1c39155401d0,c7474e00a6885e26] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:56.391 INFO 3009027 --- [manual-subtask-executor7,73ac1c39155401d0,c7474e00a6885e26] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=8, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:56.367+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=8}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:56.411 INFO 3009027 --- [manual-subtask-executor7,73ac1c39155401d0,c7474e00a6885e26] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 8, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:29:56.413 INFO 3009027 --- [manual-subtask-executor7,73ac1c39155401d0,c7474e00a6885e26] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[7] 2025-04-08 20:29:57.022 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,6c2312305b08e2b9,0d8004801ab85240] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=6c2312305b08e2b9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:57.032 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,6c2312305b08e2b9,0d8004801ab85240] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=6c2312305b08e2b9, duration=14 ms] 2025-04-08 20:29:57.101 WARN 3009027 --- [ocp-async-7,02c12a7964a6ae25,931edf8e6d2144b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:57.101 INFO 3009027 --- [ocp-async-4,2a3c6c0349d3a7af,f54e6d24c3d646ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:57.383 INFO 3009027 --- [subtask-coordinator1,1308668168866431,c68391c4080d966b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7, operation:EXECUTE, state: RUNNING 2025-04-08 20:29:57.481 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:29:57.492 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:29:57.381+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=7}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:29:57.496 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 20:29:57.507 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:57.540 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.569 INFO 3009027 --- [prometheus-minute-schedule-1,de6026b6fc748f17,e7e82afa5cc32ba1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115400, interval=60 2025-04-08 20:29:57.580 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:57.607 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.622 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:57.660 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.662 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.700 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:57.720 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.750 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:57.778 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.781 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.818 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:57.838 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.858 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:57.886 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:57.952 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.039 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:58.101 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.135 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61 2025-04-08 20:29:58.175 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:29:58.217 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'pgrep '"'"'ocp_agentd'"'"' | xargs --no-run-if-empty kill -9', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:29:58.222 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:58.242 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.259 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,bb89fe411d6f6e59,b9d62e21659118ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=bb89fe411d6f6e59, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:58.258 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:58.278 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,bb89fe411d6f6e59,b9d62e21659118ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=bb89fe411d6f6e59, duration=23 ms] 2025-04-08 20:29:58.286 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.379 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.465 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:58.524 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.658 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:58.689 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.755 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.758 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:58.776 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.781 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:58.818 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.823 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.858 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:58.880 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.884 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:58.919 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.922 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:58.959 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:59.061 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.065 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:59.093 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.101 WARN 3009027 --- [ocp-async-6,6093a0b7b059d6cc,2b7dd17f10cbc65d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:29:59.101 INFO 3009027 --- [ocp-async-3,f6171baae6dfa18b,c6fd7a4a492dda42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:29:59.097 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.133 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:59.158 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.233 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:29:59.233 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:29:59.207 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:59.278 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.322 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.332 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:59.352 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.358 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:59.385 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.392 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.425 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:59.443 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.461 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:29:59.565 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,69cf9bf42d3963ff,ef95159eff30f27f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=69cf9bf42d3963ff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:29:59.576 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,69cf9bf42d3963ff,ef95159eff30f27f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=69cf9bf42d3963ff, duration=14 ms] 2025-04-08 20:29:59.832 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.894 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.897 WARN 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:29:59.899 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:29:59.901 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:29:59.919 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.921 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:29:59.965 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:29:59.967 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.006 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:00.040 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.059 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:30:00.088 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.116 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.129 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:00.157 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.259 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61 2025-04-08 20:30:00.302 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:00.399 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'pgrep '"'"'ocp_mgragent'"'"' | xargs --no-run-if-empty kill -9', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:00.485 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:00.503 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.505 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:30:00.531 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.536 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.569 INFO 3009027 --- [prometheus-minute-schedule-1,f83e3d54a85550b5,c79fe9aececd4ad9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:00.571 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:00.593 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.598 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61 2025-04-08 20:30:00.639 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:00.644 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'pgrep '"'"'ocp_monagent'"'"' | xargs --no-run-if-empty kill -9', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:00.646 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 20:30:00.680 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:00.727 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:25:00.0, nextRun=2025-04-08 20:30:00.0, currentDate=Tue Apr 08 20:30:00 CST 2025 2025-04-08 20:30:00.727 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:25:00.0, nextRunTime=2025-04-08 20:30:00.0} 2025-04-08 20:30:00.733 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 20:30:00.733 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:30:00.733+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:30:00.741 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:25:01.0, nextRun=2025-04-08 20:30:00.0, currentDate=Tue Apr 08 20:30:00 CST 2025 2025-04-08 20:30:00.741 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:25:01.0, nextRunTime=2025-04-08 20:30:00.0} 2025-04-08 20:30:00.740 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:00.747 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 20:30:00.753 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 20:30:00.753 INFO 3009027 --- [task-coordinator1,1308668168866431,f606c7f11217c70b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 20:30:00.744 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:30:00.782 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,0b5c0f93ddfc7a97,610fd64b8ae24749] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=0b5c0f93ddfc7a97, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:00.793 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,0b5c0f93ddfc7a97,610fd64b8ae24749] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=0b5c0f93ddfc7a97, duration=16 ms] 2025-04-08 20:30:01.102 WARN 3009027 --- [ocp-async-5,416a1b1893c27a26,28fda57dd9f4a01e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:01.102 INFO 3009027 --- [ocp-async-9,56094fb986437042,2d047451f0f99f3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:01.138 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.166 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.168 WARN 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:30:01.213 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:30:01.258 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 20:30:01.261 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:01.281 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.353 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:30:01.475 INFO 3009027 --- [subtask-coordinator1,1308668168866431,8d92bcce532dab26] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 41, operation:EXECUTE, state: RUNNING 2025-04-08 20:30:01.500 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:30:01.501 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=41, context=Context{parallelIdx=-1, stringMap={task_instance_id=3, task_operation=execute, latest_execution_start_time=2025-04-08T20:30:01.473+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=41}, listMap={}}, executor=192.168.1.61 2025-04-08 20:30:01.507 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.511 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 20:30:01.514 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.514 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.515 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.516 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.517 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.518 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.518 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:30:01.528 INFO 3009027 --- [subtask-executor2,1308668168866431,0ef969673399d0e9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 41, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:30:01.726 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.729 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.767 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:01.784 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.839 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: mkdir -p /tmp/f372bda441154f3c on 192.168.1.61 2025-04-08 20:30:01.866 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: mkdir -p /tmp/f372bda441154f3c on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=mkdir -p /tmp/f372bda441154f3c, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.870 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: mkdir -p /tmp/f372bda441154f3c on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=mkdir -p /tmp/f372bda441154f3c, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.908 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:01.944 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:01.962 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: touch /tmp/f372bda441154f3c/uninstall_package.sh && chmod +x /tmp/f372bda441154f3c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package $1' > /tmp/f372bda441154f3c/uninstall_package.sh on 192.168.1.61 2025-04-08 20:30:02.028 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,3ea3cab9749df247,98ff2d93dd88388e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=3ea3cab9749df247, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:02.039 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,3ea3cab9749df247,98ff2d93dd88388e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=3ea3cab9749df247, duration=15 ms] 2025-04-08 20:30:02.050 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:3, status:SUCCESS 2025-04-08 20:30:02.053 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:3, status:SUCCESS 2025-04-08 20:30:02.024 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: touch /tmp/f372bda441154f3c/uninstall_package.sh && chmod +x /tmp/f372bda441154f3c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package $1' > /tmp/f372bda441154f3c/uninstall_package.sh on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=touch /tmp/f372bda441154f3c/uninstall_package.sh && chmod +x /tmp/f372bda441154f3c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package $1' > /tmp/f372bda441154f3c/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:02.057 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: touch /tmp/f372bda441154f3c/uninstall_package.sh && chmod +x /tmp/f372bda441154f3c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package $1' > /tmp/f372bda441154f3c/uninstall_package.sh on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=touch /tmp/f372bda441154f3c/uninstall_package.sh && chmod +x /tmp/f372bda441154f3c/uninstall_package.sh && echo '#!/bin/bash ... uninstall_package $1' > /tmp/f372bda441154f3c/uninstall_package.sh, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:02.060 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.executor.executor.SshExecutor : touch uninstall script complete, rpmName=ocp-agent-ce, result= 2025-04-08 20:30:02.066 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:02.084 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:02.094 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: /tmp/f372bda441154f3c/uninstall_package.sh 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:30:02.233 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:02.234 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:02.979 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: /tmp/f372bda441154f3c/uninstall_package.sh 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=/tmp/f372bda441154f3c/uninstall_package.sh 'ocp-agent-ce', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ 警告：/home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/basic_auth.yaml saved as /home/admin/ocp_agent/conf/config_properties/basic_auth.yaml.rpmsave , extOut=null, exitStatus=0) 2025-04-08 20:30:02.988 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: /tmp/f372bda441154f3c/uninstall_package.sh 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=/tmp/f372bda441154f3c/uninstall_package.sh 'ocp-agent-ce', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ 警告：/home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logtailer.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml saved as /home/admin/ocp_agent/conf/config_properties/ob_logcleaner.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/monagent_pipeline.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml saved as /home/admin/ocp_agent/conf/config_properties/mgragent_pipeline.yaml.rpmsave 警告：/home/admin/ocp_agent/conf/config_properties/basic_auth.yaml saved as /home/admin/ocp_agent/conf/config_properties/basic_auth.yaml.rpmsave , extOut=null, exitStatus=0) 2025-04-08 20:30:03.019 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:03.038 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:03.061 INFO 3009027 --- [ocp-schedules-2,d6ceebdd2ed22758,02022196f9c1a7a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:30:03.064 INFO 3009027 --- [ocp-schedules-1,ee3bf03376934d21,6266a7411494b265] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:30:03.042 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/tmp/f372bda441154f3c/uninstall_package.sh'"'"'' on 192.168.1.61 2025-04-08 20:30:03.082 INFO 3009027 --- [ocp-async-1,39bf44cc731bbc58,eb28aa9979c1faac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:30:03.083 INFO 3009027 --- [ocp-schedules-25,32b24cc786bda678,707c8decf0a11224] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:30:03.102 WARN 3009027 --- [ocp-async-7,1f10bf48deaddd46,2342b1dca31aabc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:03.102 INFO 3009027 --- [ocp-async-2,e53e135bb16095fb,4b4b29acb66fcc05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:03.094 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/tmp/f372bda441154f3c/uninstall_package.sh'"'"'' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/tmp/f372bda441154f3c/uninstall_package.sh'"'"'', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:03.161 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/tmp/f372bda441154f3c/uninstall_package.sh'"'"'' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/tmp/f372bda441154f3c/uninstall_package.sh'"'"'', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:30:03.176289+08:00, lastProcessTime=2025-04-08T20:29:53.176104+08:00 2025-04-08 20:30:03.169 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.ocp.executor.executor.SshExecutor : uninstall complete, rpmName=ocp-agent-ce, result= 2025-04-08 20:30:03.232 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,8433afba2cb5cd8f,f97464e834d76187] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=8433afba2cb5cd8f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:03.244 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,8433afba2cb5cd8f,f97464e834d76187] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=8433afba2cb5cd8f, duration=16 ms] 2025-04-08 20:30:03.217 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:03.468 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:30:03.176289+08:00 2025-04-08 20:30:03.316 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:03.513 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61 2025-04-08 20:30:03.569 INFO 3009027 --- [prometheus-minute-schedule-1,e4d985864aa8c9a9,c4470f9ad9f12f01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:03.607 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:03.674 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:03.722 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 20:30:03.764 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:30:03.766 INFO 3009027 --- [manual-subtask-executor8,73ac1c39155401d0,65a6e51a51b9c938] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[25] 2025-04-08 20:30:04.310 INFO 3009027 --- [ocp-schedules-22,f24ab24d418223ba,8d9f24fb49c7eb78] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:30:04.414 INFO 3009027 --- [ocp-schedules-25,337bb7f5d9c3cebe,ae4b31defeeb3527] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:30:04.422 INFO 3009027 --- [ocp-schedules-25,337bb7f5d9c3cebe,ae4b31defeeb3527] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:30:04.422 INFO 3009027 --- [ocp-schedules-25,337bb7f5d9c3cebe,ae4b31defeeb3527] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:30:04.477 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,74a6156ece475f40,998eb8c97e19039f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=74a6156ece475f40, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:04.487 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,74a6156ece475f40,998eb8c97e19039f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=74a6156ece475f40, duration=14 ms] 2025-04-08 20:30:04.490 INFO 3009027 --- [subtask-coordinator1,1308668168866431,beac742d6755376a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 25, operation:EXECUTE, state: RUNNING 2025-04-08 20:30:04.613 INFO 3009027 --- [manual-subtask-executor9,73ac1c39155401d0,fa2f17dd5288140a] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:30:04.654 INFO 3009027 --- [manual-subtask-executor9,73ac1c39155401d0,fa2f17dd5288140a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=25, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:30:04.489+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=25}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:30:04.729 INFO 3009027 --- [manual-subtask-executor9,73ac1c39155401d0,fa2f17dd5288140a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 25, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:30:04.730 INFO 3009027 --- [manual-subtask-executor9,73ac1c39155401d0,fa2f17dd5288140a] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[3] 2025-04-08 20:30:05.021 INFO 3009027 --- [task-coordinator1,1308668168866431,75bc62b74658b9ee] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2025-04-08 20:30:04.0, currentDate=Tue Apr 08 20:30:05 CST 2025 2025-04-08 20:30:05.021 INFO 3009027 --- [task-coordinator1,1308668168866431,75bc62b74658b9ee] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=null, nextRunTime=2025-04-08 20:30:04.0} 2025-04-08 20:30:05.024 INFO 3009027 --- [task-coordinator1,1308668168866431,75bc62b74658b9ee] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 20:30:05.024 INFO 3009027 --- [task-coordinator1,1308668168866431,75bc62b74658b9ee] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:30:05.024+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:30:05.102 WARN 3009027 --- [ocp-async-2,9b7cbf50cf6e4b85,8c5cdc5f3c86a3ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:05.102 INFO 3009027 --- [ocp-async-8,0da45f62199ed301,eadeb304e649addf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:05.234 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:05.235 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:05.537 INFO 3009027 --- [subtask-coordinator1,1308668168866431,3aac326b0b361295] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 3, operation:EXECUTE, state: RUNNING 2025-04-08 20:30:05.567 INFO 3009027 --- [subtask-coordinator1,1308668168866431,3aac326b0b361295] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:EXECUTE, state: RUNNING 2025-04-08 20:30:05.572 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:30:05.622 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:30:05.617 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=3, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:30:05.536+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=3}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:30:05.625 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=42, context=Context{parallelIdx=-1, stringMap={task_instance_id=4, task_operation=execute, latest_execution_start_time=2025-04-08T20:30:05.565+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=42}, listMap={}}, executor=192.168.1.61 2025-04-08 20:30:05.626 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=16fe577f-15f6-4fe7-9197-bc84ec1a1dde, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1, authType=OCP-HMACSHA1)) 2025-04-08 20:30:05.639 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 20:30:05.639 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 20:30:05.639 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.640 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:30:05.641 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.641 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.642 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.642 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 20:30:05.647 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=16fe577f-15f6-4fe7-9197-bc84ec1a1dde, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1, authType=OCP-HMACSHA1)) 2025-04-08 20:30:05.648 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.657 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.657 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.658 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 20:30:05.662 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 20:30:05.662 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 20:30:05.662 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.663 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:30:05.663 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:4 2025-04-08 20:30:05.664 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.669 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.669 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.669 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 20:30:05.689 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.690 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.690 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.691 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 20:30:05.705 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.705 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.705 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.706 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 20:30:05.716 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 3, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:30:05.718 INFO 3009027 --- [manual-subtask-executor10,73ac1c39155401d0,3012889ac0938429] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[19] 2025-04-08 20:30:05.723 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,3e0231f47646b062,5184145f07604524] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=3e0231f47646b062, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:05.726 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.726 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.727 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.727 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 20:30:05.732 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,3e0231f47646b062,5184145f07604524] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=3e0231f47646b062, duration=13 ms] 2025-04-08 20:30:05.743 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.743 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.743 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.744 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 20:30:05.761 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.761 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.762 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.762 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 20:30:05.766 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.766 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.766 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.766 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 20:30:05.770 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.770 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.770 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.771 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 20:30:05.784 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.784 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.784 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.785 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 20:30:05.796 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.796 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.796 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.796 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 20:30:05.806 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.806 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.807 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.807 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 20:30:05.823 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.823 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.823 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.824 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 20:30:05.838 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.838 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.839 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.839 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 20:30:05.844 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.844 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.844 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.845 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 20:30:05.853 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.853 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.854 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.854 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 20:30:05.858 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.858 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.859 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.859 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 20:30:05.862 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.862 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.863 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.863 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 20:30:05.866 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.867 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.867 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.867 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 20:30:05.872 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.872 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.873 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.873 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 20:30:05.877 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.878 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.878 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.878 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 20:30:05.882 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.882 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.882 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.882 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 20:30:05.886 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.886 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.887 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.887 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 20:30:05.890 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.890 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.891 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.891 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 20:30:05.894 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.895 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.895 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.895 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 20:30:05.899 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.899 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.900 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.900 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 20:30:05.903 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.904 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.904 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.904 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 20:30:05.908 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.908 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.909 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.909 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 20:30:05.913 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.913 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.913 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.914 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 20:30:05.917 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.917 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.918 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.918 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 20:30:05.921 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.921 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.922 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.922 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 20:30:05.925 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.926 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.926 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.926 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 20:30:05.930 INFO 3009027 --- [ocp-schedules-4,f7dc9d2d0b9851b4,3c723e4913829205] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:30:05.930 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.930 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.931 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.931 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 20:30:05.932 INFO 3009027 --- [ocp-schedules-4,f7dc9d2d0b9851b4,3c723e4913829205] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:30:05.935 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.935 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.935 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.935 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 20:30:05.939 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.939 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.939 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.939 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 20:30:05.948 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.949 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.949 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.949 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 20:30:05.957 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.957 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.957 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.958 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 20:30:05.962 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.963 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.963 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.963 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 20:30:05.972 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.972 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.973 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.973 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 20:30:05.976 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.976 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.976 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.977 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 20:30:05.980 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.980 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.980 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.980 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 20:30:05.983 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.983 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.983 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.984 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 20:30:05.987 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.987 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.987 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.987 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 20:30:05.996 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.996 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:05.997 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:05.997 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 20:30:06.016 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.016 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:06.016 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.017 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 20:30:06.022 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.022 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:06.023 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.023 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 20:30:06.028 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.028 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:06.029 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.029 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 20:30:06.034 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.035 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:06.035 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.035 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 20:30:06.038 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.039 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:06.039 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.039 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 20:30:06.042 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.042 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:30:06.043 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:06.043 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 20:30:06.053 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 20:30:06.065 INFO 3009027 --- [subtask-executor3,1308668168866431,fd8166c9fdd3147e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 42, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:30:06.569 INFO 3009027 --- [prometheus-minute-schedule-1,58b35dd3d4fe3814,af5dd3b541c368f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:06.577 INFO 3009027 --- [subtask-coordinator1,1308668168866431,2f0499a18f571c98] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 19, operation:EXECUTE, state: RUNNING 2025-04-08 20:30:06.629 INFO 3009027 --- [manual-subtask-executor11,73ac1c39155401d0,fb45be73546746c1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:30:06.633 INFO 3009027 --- [manual-subtask-executor11,73ac1c39155401d0,fb45be73546746c1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=19, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:30:06.575+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=19}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:30:06.666 INFO 3009027 --- [manual-subtask-executor11,73ac1c39155401d0,fb45be73546746c1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 19, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:30:06.667 INFO 3009027 --- [manual-subtask-executor11,73ac1c39155401d0,fb45be73546746c1] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[26] 2025-04-08 20:30:06.935 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,4733a0df225c9c05,8647f37c7b07c709] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=4733a0df225c9c05, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:06.945 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,4733a0df225c9c05,8647f37c7b07c709] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=4733a0df225c9c05, duration=14 ms] 2025-04-08 20:30:07.079 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:4, status:SUCCESS 2025-04-08 20:30:07.081 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:4, status:SUCCESS 2025-04-08 20:30:07.102 WARN 3009027 --- [ocp-async-10,942c9b7231e0ae2e,57e146c71a623bbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:07.102 INFO 3009027 --- [ocp-async-9,4592d8803206a570,5d1b2866842731cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:07.625 INFO 3009027 --- [subtask-coordinator1,1308668168866431,e704afb7b95dfbba] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:EXECUTE, state: RUNNING 2025-04-08 20:30:07.636 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:30:07.638 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=26, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:30:07.624+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=26}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:30:07.640 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 20:30:07.651 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:30:07.660 WARN 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:30:07.664 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:30:07.713 WARN 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:30:07.755 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:30:07.766 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:30:07.777 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:07.880 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:07.927 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:07.988 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:08.029 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:30:08.102 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:08.129 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:08.135 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:08.146 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,399ab20b5a6944d3,18f8bd0a0543b579] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=399ab20b5a6944d3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:08.154 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:08.157 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,399ab20b5a6944d3,18f8bd0a0543b579] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=399ab20b5a6944d3, duration=15 ms] 2025-04-08 20:30:08.195 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:30:08.235 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:08.236 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:08.238 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:08.262 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ , extOut=null, exitStatus=0) 2025-04-08 20:30:08.274 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 20:30:08.328 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:30:08.347 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:30:08.388 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 20:30:08.450 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:30:08.453 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:30:08.456 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:/etc/sudoers:121:12: 语法错误 Same thing without a password ^~~~~~~ curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 20:30:08.471 ERROR 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 20:30:08.477 ERROR 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:30:08.534 WARN 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:30:08.547 INFO 3009027 --- [manual-subtask-executor12,73ac1c39155401d0,a9e720e1db57d44b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:EXECUTE, state: FAILED 2025-04-08 20:30:08.852 INFO 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, createTime=2025-04-08T20:29:48+08:00, startTime=2025-04-08T20:29:48.360+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=31, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=12, startTime=null, endTime=null}, SubtaskInstanceEntity{id=12, name=Attach cluster attributes, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.AttachClusterAttributesTask, seriesId=14, startTime=null, endTime=null}, SubtaskInstanceEntity{id=2, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=18, name=Update host arch info, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateHostSystemInfoTask, seriesId=7, startTime=2025-04-08T20:29:52.261+08:00, endTime=2025-04-08T20:29:52.748+08:00}, SubtaskInstanceEntity{id=3, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=16, startTime=2025-04-08T20:30:05.536+08:00, endTime=2025-04-08T 2025-04-08 20:30:08.874 WARN 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:30:08.875 INFO 3009027 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2025-04-08 20:30:09.102 WARN 3009027 --- [ocp-async-7,3c51d6cae4db2249,5915dc389f15171f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:09.102 INFO 3009027 --- [ocp-async-1,57abba7cef7d9a56,156cfd287ed19df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:09.199 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,c8585e569c790284,5ebd80be87fae557] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=c8585e569c790284, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:30:09.212 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,c8585e569c790284,5ebd80be87fae557] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=c8585e569c790284, duration=18 ms] 2025-04-08 20:30:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,1bb59c49fe2eacfa,e3b1cf747add9aac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:09.840 INFO 3009027 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:30:09.840 INFO 3009027 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:30:10.941 WARN 3009027 --- [ocp-async-6,6cd51624326ca6f1,34b9ba038d7eb3fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:10.941 INFO 3009027 --- [ocp-async-3,ba5047bafd66259f,ac2b1d07c46cd52c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:11.074 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:11.075 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:11.091 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:30:12.239 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T20:24:12.188, end=2025-04-08T20:29:12.188} 2025-04-08 20:30:12.241 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 20:30:12.241 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:30:12.281 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T20:24:12.203, end=2025-04-08T20:29:12.203} 2025-04-08 20:30:12.283 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 20:30:12.284 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:30:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,cf385efbb072e894,2045c6ec29de1c10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:12.408 INFO 3009027 --- [prometheus-cache-monitor1,287825556bf037b5,8f8aa04f2ae7011a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1108,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":416,"secondWriteQueue":0,"minuteWriteCacheSize":403} 2025-04-08 20:30:12.408 INFO 3009027 --- [prometheus-cache-monitor1,287825556bf037b5,8f8aa04f2ae7011a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=61, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=61, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:30:12.920 INFO 3009027 --- [ocp-async-8,2bc67e6243195499,121ff1e88917ba7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:30:12.923 INFO 3009027 --- [ocp-schedules-6,912a3c523f1a118e,46db0a54b8d98cec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:30:12.941 WARN 3009027 --- [ocp-async-10,92be0c9cc911b0f4,e9f277490d9acb1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:12.941 INFO 3009027 --- [ocp-async-9,5270d4db711f487c,efc6f05da528f021] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:30:13.176933+08:00, lastProcessTime=2025-04-08T20:30:03.176289+08:00 2025-04-08 20:30:13.262 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:30:13.176933+08:00 2025-04-08 20:30:14.075 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:14.076 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:14.177 INFO 3009027 --- [ocp-schedules-13,148e83aed183822c,a9bec996842ccb46] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:30:14.941 WARN 3009027 --- [ocp-async-7,12b6f61dec534b04,5c39a2f98f9bd558] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:14.941 INFO 3009027 --- [ocp-async-4,ab2a815f755ee797,e30b659a4587c46c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,0c12c55aa1e30379,30e89e5fcaee30c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:15.514 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=48, affectRows=48 2025-04-08 20:30:15.514 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=48, validCount=48 2025-04-08 20:30:15.516 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=42, affectRows=42 2025-04-08 20:30:15.516 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=44, affectRows=44 2025-04-08 20:30:15.516 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=71, affectRows=71 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=41, validCount=41 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=44, validCount=44 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=48, affectRows=48 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=90, affectRows=90 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=69, validCount=69 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115340, nodeCount=73, affectRows=73 2025-04-08 20:30:15.517 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=48, validCount=48 2025-04-08 20:30:15.518 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=73, validCount=73 2025-04-08 20:30:15.518 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115340, nodeCount=90, validCount=90 2025-04-08 20:30:15.777 INFO 3009027 --- [ocp-schedules-3,26b7d519590d4e20,d5586e64d13b46bd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:30:15.778 INFO 3009027 --- [ocp-schedules-3,26b7d519590d4e20,d5586e64d13b46bd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:30:16.941 WARN 3009027 --- [ocp-async-6,34aeffd82cacc252,c8c19572623abfa5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:16.941 INFO 3009027 --- [ocp-async-2,ed149f2cabd25827,c9b87fabf5c8a60b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:17.076 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:17.076 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,d845c807849ffdef,eafe31cdaad34f22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:18.941 WARN 3009027 --- [ocp-async-10,da3b5e45f64449e7,f4837d7dc706edea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:18.941 INFO 3009027 --- [ocp-async-5,416dfb0b0da20047,1123d31febd4534f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:20.077 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:20.077 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:20.941 WARN 3009027 --- [ocp-async-7,2a647a1f9523c1b5,b5cefca31ca70c4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:20.941 INFO 3009027 --- [ocp-async-5,35d9979cb906608d,53bfdfd60a5d8687] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,e7524ab742290a19,79f6b3b1df3638ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:22.895 INFO 3009027 --- [ocp-schedules-5,2133332102c6bff4,12b3d769dea5af84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:30:22.896 INFO 3009027 --- [ocp-schedules-1,310d84c6c2ce8538,ae4c05b61dc16929] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:30:22.896 INFO 3009027 --- [ocp-schedules-10,7898c690a5bd3f66,5f43a0f659082ccd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:30:22.896 INFO 3009027 --- [ocp-schedules-27,fc3e13e364e03dd8,a01abb75956022d2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:30:22.896 INFO 3009027 --- [ocp-schedules-3,f5eb2bbe3206495b,eb9227ccbc00d668] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:30:22.897 INFO 3009027 --- [ocp-schedules-10,7898c690a5bd3f66,5f43a0f659082ccd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.897 INFO 3009027 --- [ocp-schedules-3,f5eb2bbe3206495b,eb9227ccbc00d668] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:30:22.898 INFO 3009027 --- [ocp-schedules-19,9aebfca473b124f4,d87dc60141754788] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:30:22.898 INFO 3009027 --- [ocp-schedules-7,31f78fbb2a834581,18ca91313381eebe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:30:22.898 INFO 3009027 --- [ocp-schedules-30,4ef675be0b7fff83,a983ae0d101de95b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:30:22.898 INFO 3009027 --- [ocp-schedules-30,4ef675be0b7fff83,a983ae0d101de95b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:30:22.899 INFO 3009027 --- [ocp-schedules-7,31f78fbb2a834581,18ca91313381eebe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:30:22.899 INFO 3009027 --- [ocp-schedules-19,9aebfca473b124f4,d87dc60141754788] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:30:22.899 INFO 3009027 --- [ocp-schedules-5,2133332102c6bff4,12b3d769dea5af84] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:30:22.900 INFO 3009027 --- [ocp-schedules-13,9888ed12c0159c27,68a2a23e0da08380] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:30:22.903 INFO 3009027 --- [ocp-schedules-27,fc3e13e364e03dd8,a01abb75956022d2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:30:22.903 INFO 3009027 --- [ocp-schedules-28,419c25eceb7c9c4b,a1ad57eac6f7f682] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:30:22.903+08:00, clusters=[1] 2025-04-08 20:30:22.904 INFO 3009027 --- [ocp-schedules-6,690fbba1094818aa,82c30c905262a751] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:30:22.904+08:00, clusters=[1] 2025-04-08 20:30:22.910 INFO 3009027 --- [ocp-schedules-6,690fbba1094818aa,82c30c905262a751] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:30:22.912 INFO 3009027 --- [ocp-schedules-28,419c25eceb7c9c4b,a1ad57eac6f7f682] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:30:22.913 INFO 3009027 --- [ocp-schedules-29,4d7ec2e7f99a4d15,eabd4a1da1da2f8c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1524,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":414,"seriesIdKeySize":416,"secondWriteQueue":0,"minuteWriteCacheSize":413} 2025-04-08 20:30:22.913 INFO 3009027 --- [ocp-schedules-29,4d7ec2e7f99a4d15,eabd4a1da1da2f8c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=61, ob_cluster_status_check=7, system_cpu_count=2, jvm_gc_pause_seconds_sum=7, jvm_threads_states_threads=8, ocp_alarm_detect_duration_ms_sum=134, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, jvm_threads_live_threads=2, ocp_meta_db_health=4, system_load_average_1m=2, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, http_server_requests_seconds_count=61, jvm_gc_memory_allocated_bytes_total=2, jvm_gc_pause_seconds_count=7, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_gc_overhead_percent=2, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10} 2025-04-08 20:30:22.920 INFO 3009027 --- [ocp-schedules-22,85e9c6c9b18b616f,070cf1b923b07027] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:30:22.920 INFO 3009027 --- [ocp-async-3,204d51deba8279b1,610c1a30b5b76a7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:30:22.921 INFO 3009027 --- [ocp-schedules-22,85e9c6c9b18b616f,070cf1b923b07027] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.921 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:30:22.921 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:30:22.921 INFO 3009027 --- [ocp-schedules-25,41f7e12ed1ab2cf7,959bca066a0104d8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:30:22.922 INFO 3009027 --- [ocp-schedules-25,41f7e12ed1ab2cf7,959bca066a0104d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.924 INFO 3009027 --- [ocp-schedules-9,2a640b1609472209,a6c61f8cdfd6e4fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:30:22.926 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:30:22.927 WARN 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:30:22.927 INFO 3009027 --- [ocp-schedules-16,e75098d5526e3e55,e47468a48261fec0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:30:22.927 INFO 3009027 --- [ocp-schedules-16,7419fe49ad243170,01c2064bf896d584] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:30:22.927 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:30:22.928 INFO 3009027 --- [ocp-schedules-17,d697b688140c2983,8f35778ac477c795] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:30:22.928 WARN 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:30:22.928 INFO 3009027 --- [ocp-schedules-16,7419fe49ad243170,01c2064bf896d584] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.928 INFO 3009027 --- [ocp-schedules-17,d697b688140c2983,8f35778ac477c795] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.930 INFO 3009027 --- [ocp-schedules-1,f5f8818b11d0e838,3c415b78c703b6aa] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:30:22.931 INFO 3009027 --- [ocp-schedules-1,f5f8818b11d0e838,3c415b78c703b6aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:22.931 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:30:22.935 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:30:22.935 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:30:22.935 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.e.internal.connector.Connectors : init global executor service. 2025-04-08 20:30:22.942 WARN 3009027 --- [ocp-async-4,aea57e357513b67f,9ded53023ff89e53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:22.942 INFO 3009027 --- [ocp-async-4,c1d01aa62d8e6d90,28c112a739025c86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:22.943 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=ocpdb, obClusterId=1744114127, tenantName=sys, username=root, address=192.168.1.61, port=2881, database=oceanbase) 2025-04-08 20:30:22.943 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 20:30:22.945 ERROR 3009027 --- [ocp-schedules-17,d697b688140c2983,8f35778ac477c795] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:30:22.945 INFO 3009027 --- [ocp-schedules-17,d697b688140c2983,8f35778ac477c795] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:30:22.947 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:30:22.948 ERROR 3009027 --- [ocp-schedules-25,41f7e12ed1ab2cf7,959bca066a0104d8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:30:22.948 INFO 3009027 --- [ocp-schedules-25,41f7e12ed1ab2cf7,959bca066a0104d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:30:22.948 INFO 3009027 --- [ocp-schedules-27,339a31bbdeb94e76,c60d6e9b2388db5c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:30:22.949 INFO 3009027 --- [obsdk-init-thread-4,e3ca775a13cfb493,bff8b82e6602b60e] com.alibaba.druid.pool.DruidDataSource : {dataSource-6,ob-connector-datasource_direct_sys_root_192.168.1.61_2881} inited 2025-04-08 20:30:22.949 INFO 3009027 --- [obsdk-init-thread-4,e3ca775a13cfb493,bff8b82e6602b60e] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 20:30:22.949 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 20:30:22.950 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:22.951 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:30:22.952 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:22.952 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:30:22.956 INFO 3009027 --- [ocp-schedules-6,0f74721e2c7a3ffc,e8f258b8d00d57ce] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:30:22.958 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='cluster_cgroup_value', labels={app=OB, ob_cluster_id=1744114127, ob_cluster_name=ocpdb}}, value: 0, timestamp: 1744115422 : 2025-04-08 12:30:22 2025-04-08 20:30:22.960 INFO 3009027 --- [ocp-schedules-2,5e1e5dea84b6b6ae,e83718378bb0daa8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:30:22.960 INFO 3009027 --- [ocp-schedules-2,5e1e5dea84b6b6ae,e83718378bb0daa8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:30:22.965 INFO 3009027 --- [ocp-schedules-2,5e1e5dea84b6b6ae,e83718378bb0daa8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:30:22.965 INFO 3009027 --- [ocp-schedules-2,5e1e5dea84b6b6ae,e83718378bb0daa8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:30:22.970 INFO 3009027 --- [ocp-schedules-5,be0a896e9149f091,28ce33323789da78] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:30:22.970 INFO 3009027 --- [ocp-schedules-5,be0a896e9149f091,28ce33323789da78] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:30:22.989 INFO 3009027 --- [ocp-schedules-11,bb437105d0e263af,36e13733d7b09aa4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:30:22.989 INFO 3009027 --- [ocp-schedules-24,9eca0058a7e11287,72f752a033ec9981] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:30:22.990 INFO 3009027 --- [ocp-schedules-24,9eca0058a7e11287,72f752a033ec9981] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:30:22.996 INFO 3009027 --- [ocp-schedules-11,bb437105d0e263af,36e13733d7b09aa4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:30:22.996 INFO 3009027 --- [ocp-schedules-11,bb437105d0e263af,36e13733d7b09aa4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:30:23.040 INFO 3009027 --- [ocp-schedules-8,86afbf41d0f65e14,02ca65abc2329f61] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:30:23.065 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:30:23.065 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:30:23.066 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:30:23.066 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766197921368895488 2025-04-08 20:30:23.066 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=12, maxSyncedRecordId=766198172838391808 2025-04-08 20:30:23.067 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:30:23.067 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:30:23.069 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.61, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:2 2025-04-08 20:30:23.069 INFO 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:30:23.073 INFO 3009027 --- [ocp-schedules-26,dcba5dacb2e27cd7,5b6a16b0deb877d4] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=187, maxSyncedRecordId=766198242308648960 2025-04-08 20:30:23.077 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:23.078 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:23.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:30:23.177778+08:00, lastProcessTime=2025-04-08T20:30:13.176933+08:00 2025-04-08 20:30:23.280 INFO 3009027 --- [ocp-schedules-9,c9b3d564136dd378,eaa46ca88d96b37d] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,,, current targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61 2025-04-08 20:30:23.334 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:30:23.177778+08:00 2025-04-08 20:30:23.419 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 20:30:23.428 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.428 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 20:30:23.428 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 20:30:23.428 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 20:30:23.429 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 20:30:23.430 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 20:30:23.431 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 20:30:23.432 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 20:30:23.433 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 20:30:23.434 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 20:30:23.435 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 20:30:23.436 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:30:23.437 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 20:30:23.438 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 20:30:23.439 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 20:30:23.440 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 20:30:23.441 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:30:23.442 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 20:30:23.443 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:30:23.444 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 20:30:23.445 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 20:30:23.446 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 20:30:23.447 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 20:30:23.448 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 20:30:23.449 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 20:30:23.450 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 20:30:23.451 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 20:30:23.452 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 20:30:23.453 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 20:30:23.454 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 20:30:23.455 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 20:30:23.457 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 20:30:23.457 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.457 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 20:30:23.457 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 20:30:23.458 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 20:30:23.459 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 20:30:23.460 INFO 3009027 --- [ocp-schedules-4,ed00f189be9d4d6c,b75694e459774067] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 20:30:23.645 INFO 3009027 --- [ocp-schedules-7,29adcd84a6c8c049,d210d9dc45bebae5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:30:23.742 WARN 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:30:23.742 ERROR 3009027 --- [ocp-schedules-20,fdd21999d228f7a7,31bf83dc16ba8b89] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:30:24.181 INFO 3009027 --- [ocp-schedules-15,08d7796767a3436b,877a85266f9fa630] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:30:24.262 INFO 3009027 --- [ocp-schedules-23,a73b9006054f7875,28803f28b18858c4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:30:24.280 INFO 3009027 --- [ocp-schedules-23,a73b9006054f7875,28803f28b18858c4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:30:24.280 INFO 3009027 --- [ocp-schedules-23,a73b9006054f7875,28803f28b18858c4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:30:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,e91172b013352d20,e9782acf1688130e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:24.942 WARN 3009027 --- [ocp-async-9,6cdd0c6041cd6d36,5f48fbc428dec7fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:24.942 INFO 3009027 --- [ocp-async-4,9c274e8d71faa458,ad6f966f6e0aeb60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:25.649 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:30:25.649 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:30:25.652 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:30:25.652 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:30:25.652 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:30:25.652 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:30:25.656 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:30:25.656 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:30:25.656 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:30:25.656 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:30:25.680 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:30:25.685 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:30:25.685 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:30:25.685 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:30:25.685 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:30:25.685 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:30:25.691 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:30:25.780 INFO 3009027 --- [ocp-schedules-25,752423597b49c3c9,43ce189d556aef6c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:30:25.781 INFO 3009027 --- [ocp-schedules-25,752423597b49c3c9,43ce189d556aef6c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:30:26.043 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:30:26.078 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:26.079 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:26.356 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.699629621S 2025-04-08 20:30:26.356 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:30:26.412 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:30:26.412 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:30:26.440 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:30:26.485 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:30:26.485 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:30:26.485 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:30:26.485 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:30:26.485 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:30:26.485 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:30:26.492 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:30:26.592 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] o.s.boot.SpringApplication : Started application in 0.939 seconds (JVM running for 468.21) 2025-04-08 20:30:26.593 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.94019507S 2025-04-08 20:30:26.881 INFO 3009027 --- [ocp-schedules-21,c3c91e0f8b2abbee,e5cc679d3751dc3f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:30:26.942 WARN 3009027 --- [ocp-async-5,42eef75d8a45d5eb,95e893b3a6c20be3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:26.942 INFO 3009027 --- [ocp-async-6,11a22d75bf5298c2,9aac1a1b450985ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:30:27.385 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,053dcf4cc4a7bbfc,f24a7bef9b1cae80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=053dcf4cc4a7bbfc, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:30:27.389 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,053dcf4cc4a7bbfc,f24a7bef9b1cae80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=053dcf4cc4a7bbfc, duration=10 ms] 2025-04-08 20:30:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,17285a4627b30d34,c4412d950c257308] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:28.942 WARN 3009027 --- [ocp-async-8,2168f676b259b439,d6c1cb76d510c8b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:28.942 INFO 3009027 --- [ocp-async-10,1f3b724354004611,52ad00a631edbc60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:29.079 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:29.079 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,8229cfcf293403f1,5eee327f4dab0c0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:30.942 WARN 3009027 --- [ocp-async-7,d9e9b9c287c151a1,916a7b3022b1741c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:30.942 INFO 3009027 --- [ocp-async-9,af60eb4bdaddc3fa,4e12f86ed6338948] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:32.079 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:32.080 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:32.897 INFO 3009027 --- [ocp-schedules-10,7898c690a5bd3f66,5f43a0f659082ccd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:30:32.898 INFO 3009027 --- [ocp-schedules-3,f5eb2bbe3206495b,eb9227ccbc00d668] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:30:32.921 INFO 3009027 --- [ocp-async-5,b7aacb12b0a360c6,c34eba8048aff243] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:30:32.925 INFO 3009027 --- [ocp-schedules-15,53abd496b8cc6f93,621725f9c4546225] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:30:32.931 INFO 3009027 --- [ocp-schedules-1,f5f8818b11d0e838,3c415b78c703b6aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:30:32.942 WARN 3009027 --- [ocp-async-4,1949896ef4f587ac,accb704c51eeb4bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:32.942 INFO 3009027 --- [ocp-async-3,3e9318d042ee2b91,2f31319d4fe16df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:30:33.177372+08:00, lastProcessTime=2025-04-08T20:30:23.177778+08:00 2025-04-08 20:30:33.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:30:33.177372+08:00 2025-04-08 20:30:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,6ea892348f63bf5f,d4677454d5f8f9d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:34.284 INFO 3009027 --- [ocp-schedules-28,d7b012c933b18931,286d8916c55aebdd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:30:34.943 WARN 3009027 --- [ocp-async-2,73262478db4c98b3,ced813be8c20f94e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:34.943 INFO 3009027 --- [ocp-async-6,7a0f80033a4d4425,c2ffbf560785c57c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:35.080 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:35.081 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:35.898 INFO 3009027 --- [ocp-schedules-4,200ab4eaa13d435b,7543265f9f63cad5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:30:35.900 INFO 3009027 --- [ocp-schedules-4,200ab4eaa13d435b,7543265f9f63cad5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:30:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,8becf7c9abe55dd9,1c3a6332632ccec0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:36.943 WARN 3009027 --- [ocp-async-9,5c679dbf9e8feb43,9cfbf9a04b829693] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:36.943 INFO 3009027 --- [ocp-async-6,8cae99767e550d4d,ed8d06ef6e5c65f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:38.081 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:38.082 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:38.943 WARN 3009027 --- [ocp-async-8,3ce8a553e9af645c,fc17f22eedabb495] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:38.943 INFO 3009027 --- [ocp-async-6,7aee3e4b13b8b8bf,bb6bd8666cd67465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,53f27f3c8cb13b2a,659059c674a2976e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:40.001 INFO 3009027 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:30:40.001 INFO 3009027 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:30:40.943 WARN 3009027 --- [ocp-async-7,7b41ce78b2c01328,24f7c4ba6a301277] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:40.943 INFO 3009027 --- [ocp-async-6,1852a20288cc5991,b925c75207db56a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:40.969 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:40.970 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:30:40.970 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:30:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:30:40.971 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:30:40.978 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1), status=TAKINGOVER 2025-04-08 20:30:40.978 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=7ms, period=120000ms 2025-04-08 20:30:41.082 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:41.082 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,f3a9b53923906d10,3256e2fc0bfc100c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:42.921 INFO 3009027 --- [ocp-async-6,9c4214542f3753d9,edf3a33300c21a92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:30:42.926 INFO 3009027 --- [ocp-schedules-15,111f0e087ad6d752,084b53d6f1385c45] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:30:42.943 WARN 3009027 --- [ocp-async-10,e9a2a597d07f11b7,cfd549b35a0fa48b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:42.943 INFO 3009027 --- [ocp-async-9,0fe2b799f4c1e7ad,b2a4eca0d315f5f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:30:43.177113+08:00, lastProcessTime=2025-04-08T20:30:33.177372+08:00 2025-04-08 20:30:43.282 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:30:43.177113+08:00 2025-04-08 20:30:44.083 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:44.083 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:44.282 INFO 3009027 --- [ocp-schedules-24,a4854b1cd4fe8ece,a5747c0112523fdf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:30:44.286 INFO 3009027 --- [ocp-schedules-24,a4854b1cd4fe8ece,a5747c0112523fdf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:30:44.286 INFO 3009027 --- [ocp-schedules-24,a4854b1cd4fe8ece,a5747c0112523fdf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:30:44.311 INFO 3009027 --- [ocp-schedules-11,a98366c46adcefac,0d46163a38aa02d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:30:44.943 WARN 3009027 --- [ocp-async-3,56b00a53a4616690,0b411ae495df5882] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:44.943 INFO 3009027 --- [ocp-async-9,52cb6c45a2f4f9f9,183a712357f910df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,336e2609fcb497e9,4e12af8aaf276276] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:45.906 INFO 3009027 --- [ocp-schedules-18,d1428ca076154cd0,a25ab11ea3601ce9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:30:45.908 INFO 3009027 --- [ocp-schedules-18,d1428ca076154cd0,a25ab11ea3601ce9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:30:46.943 WARN 3009027 --- [ocp-async-2,0ffe3b73ceb21055,62064774c9c0b51e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:46.943 INFO 3009027 --- [ocp-async-1,f64ee90c7a8c65ae,388d8aeece7edd24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:47.083 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:47.084 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,b59098df058dea0f,bfdad5f51e8a91f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:48.944 WARN 3009027 --- [ocp-async-4,75db892768317bef,918882e433932290] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:48.944 INFO 3009027 --- [ocp-async-6,d4b5f5e9444eb3cd,e5a8c0cc4de1948b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:50.084 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:50.085 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:50.944 WARN 3009027 --- [ocp-async-3,18f824a92934eb58,b93a07b575cb33a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:50.944 INFO 3009027 --- [ocp-async-6,962d28f03bbe1d1a,9b1e12403a8eb70e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,951dbd1fe068b786,8af61f64c52a4ed0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:52.896 INFO 3009027 --- [ocp-schedules-30,9d90ebc541664ab4,096ea513e7c3a060] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:30:52.896 INFO 3009027 --- [ocp-schedules-15,7ccc14e496eeca9c,225d9961db8c4175] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:30:52.896 INFO 3009027 --- [ocp-schedules-7,1c8f886dbc43f71f,802e65af47d2a71f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:30:52.897 INFO 3009027 --- [ocp-schedules-15,7ccc14e496eeca9c,225d9961db8c4175] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:52.897 INFO 3009027 --- [ocp-schedules-7,1c8f886dbc43f71f,802e65af47d2a71f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:30:52.899 INFO 3009027 --- [ocp-schedules-24,17a96d81483a9f27,d962adb137dd5a6c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:30:52.899 INFO 3009027 --- [ocp-schedules-17,cbcaf28c4172e991,fc75f02ee25164cc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:30:52.899 INFO 3009027 --- [ocp-schedules-24,17a96d81483a9f27,d962adb137dd5a6c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:30:52.899 INFO 3009027 --- [ocp-schedules-17,cbcaf28c4172e991,fc75f02ee25164cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:30:52.899 INFO 3009027 --- [ocp-schedules-3,aa963a003144dd0b,290f9ff4523d980b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:30:52.899 INFO 3009027 --- [ocp-schedules-3,aa963a003144dd0b,290f9ff4523d980b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:30:52.921 INFO 3009027 --- [ocp-async-6,863cc8cb05fc566d,3b56f09c6cf68bd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:30:52.928 INFO 3009027 --- [ocp-schedules-27,f96ae76c1acf0df7,be0d86e755104330] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:30:52.935 INFO 3009027 --- [ocp-schedules-16,7419fe49ad243170,01c2064bf896d584] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:30:52.935 INFO 3009027 --- [ocp-schedules-16,8565c87f7c6763b9,cf4a4e55774c2051] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:30:52.936 INFO 3009027 --- [ocp-schedules-16,8565c87f7c6763b9,cf4a4e55774c2051] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:30:52.941 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:30:52.941 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:30:52.944 WARN 3009027 --- [ocp-async-2,23bad08e2e18378b,bdb0decc4153d467] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:52.944 INFO 3009027 --- [ocp-async-10,77bf89e40a3a0bd7,1098c7f28442249f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:52.957 INFO 3009027 --- [ocp-schedules-29,1b0a626c98655604,9d17d07b13b936db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:30:52.964 INFO 3009027 --- [ocp-schedules-8,01580bec041d3efc,177039dbef13c01a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:30:53.085 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:53.086 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:30:53.177431+08:00, lastProcessTime=2025-04-08T20:30:43.177113+08:00 2025-04-08 20:30:53.301 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:30:53.177431+08:00 2025-04-08 20:30:54.357 INFO 3009027 --- [ocp-schedules-6,d755a37650ca0993,3bfad13e3f8e91cf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:30:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,ee3a2cb77a4bceea,f09ebef5b283d1f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:54.944 WARN 3009027 --- [ocp-async-1,8694d6412b597225,8e0c4c0e4f3475fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:54.944 INFO 3009027 --- [ocp-async-10,090b5ff0a33ce69e,6765ac4e46629302] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:55.912 INFO 3009027 --- [ocp-schedules-18,f8411a55f4fd1144,f5350a0b715a6111] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:30:55.913 INFO 3009027 --- [ocp-schedules-18,f8411a55f4fd1144,f5350a0b715a6111] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:30:56.086 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:56.087 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:30:56.944 WARN 3009027 --- [ocp-async-7,bbf6935c4edf9f03,4b2559f9a364dc87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:56.944 INFO 3009027 --- [ocp-async-10,baa4ff8117ee21ea,f791e4208854241a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,8e3ff32ac13117fc,53e5b4662411de1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115460, interval=60 2025-04-08 20:30:58.944 WARN 3009027 --- [ocp-async-9,a63b0f1a59f7be99,5cb864d43226dd6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:30:58.944 INFO 3009027 --- [ocp-async-10,9d224bc85f4fabdd,c94d8e14431f229e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:30:59.087 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:30:59.087 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,93285b99d52fb4b1,3cf5ec821193da51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:00.945 WARN 3009027 --- [ocp-async-3,ac4730fceac6dd42,390019866b0195eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:00.945 INFO 3009027 --- [ocp-async-1,03bfb8a5c50b5238,de3a779ee6778c52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:02.087 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:02.089 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:02.897 INFO 3009027 --- [ocp-schedules-15,7ccc14e496eeca9c,225d9961db8c4175] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:31:02.898 INFO 3009027 --- [ocp-schedules-7,1c8f886dbc43f71f,802e65af47d2a71f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:31:02.921 INFO 3009027 --- [ocp-async-10,e73825a20293c31e,6cdb7251041aa3e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:31:02.928 INFO 3009027 --- [ocp-schedules-20,b363a0a19b6d2fe4,bcdd7a885fc7da1c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:31:02.945 WARN 3009027 --- [ocp-async-5,afebe0fbdccde9d7,c4b3b07711c1a90d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:02.945 INFO 3009027 --- [ocp-async-1,2ade0c28cfcca0dc,90f7d2d3f0408b57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:31:03.177494+08:00, lastProcessTime=2025-04-08T20:30:53.177431+08:00 2025-04-08 20:31:03.310 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:31:03.177494+08:00 2025-04-08 20:31:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,32706538cc11d2bd,4e06d02ff841c693] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:04.287 INFO 3009027 --- [ocp-schedules-5,cb4aa3805b5eaaa2,5948c49fc6e385fd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:31:04.291 INFO 3009027 --- [ocp-schedules-5,cb4aa3805b5eaaa2,5948c49fc6e385fd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:31:04.291 INFO 3009027 --- [ocp-schedules-5,cb4aa3805b5eaaa2,5948c49fc6e385fd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:31:04.381 INFO 3009027 --- [ocp-schedules-9,7fc9afc29342828d,662d8c687fe0d2e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:31:04.945 WARN 3009027 --- [ocp-async-4,9593ef1150d805ef,caa2889d4efee3a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:04.945 INFO 3009027 --- [ocp-async-6,7a8379bb85ada9d0,799ba8f3b56d0211] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:05.089 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:05.090 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:05.935 INFO 3009027 --- [ocp-schedules-10,8042fa6e8b6038dd,69284d40172b05ef] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:31:05.936 INFO 3009027 --- [ocp-schedules-10,8042fa6e8b6038dd,69284d40172b05ef] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:31:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,63c3563d7f252b44,a1f4199cb3807b3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:06.945 WARN 3009027 --- [ocp-async-7,20f81e33f47fcf17,ca0aef6412506349] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:06.945 INFO 3009027 --- [ocp-async-6,d31e30a7f7d61975,6e0bae4b52b4eadf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:08.090 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:08.090 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:08.945 WARN 3009027 --- [ocp-async-5,c68b405e294cf816,b3ee0e44aceaad9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:08.945 INFO 3009027 --- [ocp-async-6,599060d7298d4c56,6c23ef9982539583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,7084c540da7d0147,c6b0274551f17795] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:10.002 INFO 3009027 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:31:10.002 INFO 3009027 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:31:10.945 WARN 3009027 --- [ocp-async-4,8b75c44f5cac3cb7,f1b18dedaae1bd60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:10.945 INFO 3009027 --- [ocp-async-6,2c6d3c4acc5007de,ebb28e0c8d3786d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:10.993 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:31:11.091 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:11.091 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,1a86fcd3f812a270,d233ef3ea27f4a24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:12.408 INFO 3009027 --- [prometheus-cache-monitor1,aa7d7e5a1d1e3d27,296df7c30587a79c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1524,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":417,"secondWriteQueue":0,"minuteWriteCacheSize":414} 2025-04-08 20:31:12.408 INFO 3009027 --- [prometheus-cache-monitor1,aa7d7e5a1d1e3d27,296df7c30587a79c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=61, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=61, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:31:12.802 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:27:12.719, end=2025-04-08T20:29:12.756} 2025-04-08 20:31:12.807 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:31:12.807 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:31:12.921 INFO 3009027 --- [ocp-async-6,d428b12f63cc0fdf,d7466423a75b3c48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:31:12.929 INFO 3009027 --- [ocp-schedules-11,cacfc04afb2e0a0d,798d8cccd73c56cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:31:12.945 WARN 3009027 --- [ocp-async-8,c99aef30909872b3,5a33e88a0003951a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:12.946 INFO 3009027 --- [ocp-async-7,7efc94c77ff74ed1,f84075432a79491b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:31:13.177017+08:00, lastProcessTime=2025-04-08T20:31:03.177494+08:00 2025-04-08 20:31:13.261 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:31:13.177017+08:00 2025-04-08 20:31:14.091 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:14.092 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:14.417 INFO 3009027 --- [ocp-schedules-21,350e09ef1b902c2e,6eb4eb43dceeb8d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:31:14.946 WARN 3009027 --- [ocp-async-2,a39ddfeaa30059c9,323358d95f5e13af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:14.946 INFO 3009027 --- [ocp-async-7,9f347fe2fd0f8db8,87e82da9310734c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,a1c2f00795c5fe53,5f1ad0558fed6f6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:15.553 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115400, nodeCount=137, affectRows=137 2025-04-08 20:31:15.555 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115400, nodeCount=136, validCount=136 2025-04-08 20:31:15.639 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115400, nodeCount=136, affectRows=136 2025-04-08 20:31:15.639 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115400, nodeCount=143, affectRows=143 2025-04-08 20:31:15.640 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115400, nodeCount=135, validCount=135 2025-04-08 20:31:15.640 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115400, nodeCount=142, validCount=142 2025-04-08 20:31:15.942 INFO 3009027 --- [ocp-schedules-21,f2d6975f3d735de5,785ceb17e87e9ef1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:31:15.943 INFO 3009027 --- [ocp-schedules-21,f2d6975f3d735de5,785ceb17e87e9ef1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:31:16.946 WARN 3009027 --- [ocp-async-7,db5ee58aa90e1024,4509df145cbd55d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:16.946 INFO 3009027 --- [ocp-async-1,db0a42e36934219c,543faf701bc988ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:17.092 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:17.093 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,e9beac05da6d3d1c,bf5d61132bb636bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:18.946 WARN 3009027 --- [ocp-async-8,8069a326bad47ed9,8e04b2ec20658a00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:18.946 INFO 3009027 --- [ocp-async-10,d3bfba872b1c016b,52ae81c4f7e51c76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:20.093 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:20.093 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:20.946 WARN 3009027 --- [ocp-async-4,941ed988a62ddee2,577cd33630b00207] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:20.946 INFO 3009027 --- [ocp-async-2,f0b473c2b38acaf1,6398982c2a475089] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,af030cce3dc5f2b5,b44fc031db1212eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:22.887 INFO 3009027 --- [ocp-schedules-10,9aae7edd2b2e31e1,08c9ca9395109251] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T20:31:22.887+08:00, tenantSize=0 2025-04-08 20:31:22.895 INFO 3009027 --- [ocp-schedules-4,df1a0b94781890b2,fdc22e1bb4bea1c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:31:22.896 INFO 3009027 --- [ocp-schedules-25,e8bccff3bd1b55bd,c54982d4d94cecbe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:31:22.896 INFO 3009027 --- [ocp-schedules-3,79ecd1367049a54a,71847fa55ed7d99f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:31:22.896 INFO 3009027 --- [ocp-schedules-26,d6ac43392637ade0,60ed4e194b96929d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:31:22.896 INFO 3009027 --- [ocp-schedules-14,a853812393122d57,5d5bcf4d135bb8bb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:31:22.897 INFO 3009027 --- [ocp-schedules-3,79ecd1367049a54a,71847fa55ed7d99f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:22.897 INFO 3009027 --- [ocp-schedules-14,a853812393122d57,5d5bcf4d135bb8bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:31:22.898 INFO 3009027 --- [ocp-schedules-21,76da834cdfc62f3a,9351271f4c1af0c8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:31:22.898 INFO 3009027 --- [ocp-schedules-21,76da834cdfc62f3a,9351271f4c1af0c8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:31:22.898 INFO 3009027 --- [ocp-schedules-18,8c02af00869da446,757e17f9de86c86a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:31:22.898 INFO 3009027 --- [ocp-schedules-18,8c02af00869da446,757e17f9de86c86a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:31:22.898 INFO 3009027 --- [ocp-schedules-6,e984e9dc4b72f47e,98741fb7f984c0fb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:31:22.898 INFO 3009027 --- [ocp-schedules-6,e984e9dc4b72f47e,98741fb7f984c0fb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:31:22.903 INFO 3009027 --- [ocp-schedules-26,d6ac43392637ade0,60ed4e194b96929d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:31:22.905 INFO 3009027 --- [ocp-schedules-11,c2578dcfe3b0fb70,e285f6412ecbd354] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:31:22.905 INFO 3009027 --- [ocp-schedules-4,df1a0b94781890b2,fdc22e1bb4bea1c9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:31:22.915 INFO 3009027 --- [ocp-schedules-5,7d65ab87f30c1e8f,da6485454d1877af] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:31:22.915+08:00, clusters=[1] 2025-04-08 20:31:22.916 INFO 3009027 --- [ocp-schedules-19,dc73315d0ca98718,4aada8a9141a86fb] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:31:22.916 INFO 3009027 --- [ocp-schedules-19,dc73315d0ca98718,4aada8a9141a86fb] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:31:22.917 INFO 3009027 --- [ocp-schedules-5,7d65ab87f30c1e8f,da6485454d1877af] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:31:22.917 INFO 3009027 --- [ocp-schedules-2,99f2a88a44253635,a1fac4ace3b9da00] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:31:22.917+08:00, clusters=[1] 2025-04-08 20:31:22.921 INFO 3009027 --- [ocp-async-2,5cbd628e8aaa3afc,0794fdf17167df66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:31:22.930 INFO 3009027 --- [ocp-schedules-2,99f2a88a44253635,a1fac4ace3b9da00] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:31:22.930 INFO 3009027 --- [ocp-schedules-17,1576afda22912a81,87cd553eca7f584a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:31:22.941 INFO 3009027 --- [ocp-schedules-16,8565c87f7c6763b9,cf4a4e55774c2051] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:31:22.941 INFO 3009027 --- [ocp-schedules-16,05ae3e0baac0aaa2,f4953da879d19dc2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:31:22.942 INFO 3009027 --- [ocp-schedules-16,05ae3e0baac0aaa2,f4953da879d19dc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:22.946 WARN 3009027 --- [ocp-async-1,e87037d6c3276fe4,6a6f2e9a891d8549] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:22.946 INFO 3009027 --- [ocp-async-6,beecbdbdccc608c4,908f2f90f2ef9bb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:22.947 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:31:22.947 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:31:22.948 INFO 3009027 --- [ocp-schedules-21,870bb9cfdedcaffc,d0cdc80b35c5565c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:31:22.948 INFO 3009027 --- [ocp-schedules-21,870bb9cfdedcaffc,d0cdc80b35c5565c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:22.949 INFO 3009027 --- [ocp-schedules-18,197bf232dc55dafa,5a2d86b56d453224] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:31:22.958 INFO 3009027 --- [ocp-schedules-22,85e9c6c9b18b616f,070cf1b923b07027] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:31:22.959 INFO 3009027 --- [ocp-schedules-26,d563d16e2fe57f0a,060d74310b829e7a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:31:22.964 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:31:22.965 ERROR 3009027 --- [ocp-schedules-21,870bb9cfdedcaffc,d0cdc80b35c5565c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:31:22.965 INFO 3009027 --- [ocp-schedules-21,870bb9cfdedcaffc,d0cdc80b35c5565c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:31:22.965 INFO 3009027 --- [ocp-schedules-11,7e497a42f398e187,653b18acd61f0675] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:31:22.965 INFO 3009027 --- [ocp-schedules-11,7e497a42f398e187,653b18acd61f0675] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:31:22.970 INFO 3009027 --- [ocp-schedules-11,7e497a42f398e187,653b18acd61f0675] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:31:22.970 INFO 3009027 --- [ocp-schedules-11,7e497a42f398e187,653b18acd61f0675] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:31:22.972 INFO 3009027 --- [ocp-schedules-15,6094c57c97e8c730,5ced359889135fcc] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:31:22.972 INFO 3009027 --- [ocp-schedules-15,6094c57c97e8c730,5ced359889135fcc] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:31:22.993 INFO 3009027 --- [ocp-schedules-4,c46d727ec0067eb7,abc723cc34638f7f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:31:22.993 INFO 3009027 --- [ocp-schedules-4,c46d727ec0067eb7,abc723cc34638f7f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:31:22.997 INFO 3009027 --- [ocp-schedules-19,a35ca87e1af6c8f5,01e4763efeea782c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:31:22.998 INFO 3009027 --- [ocp-schedules-19,a35ca87e1af6c8f5,01e4763efeea782c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:31:22.998 INFO 3009027 --- [ocp-schedules-19,a35ca87e1af6c8f5,01e4763efeea782c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:31:23.046 INFO 3009027 --- [ocp-schedules-29,69c9ad72ceaf2cdd,94f05bc4bf238ca3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:31:23.074 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:31:23.074 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:31:23.077 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:31:23.077 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198172838391808 2025-04-08 20:31:23.080 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=1, maxSyncedRecordId=766198297748959232 2025-04-08 20:31:23.080 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:31:23.080 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198242308648960 2025-04-08 20:31:23.086 INFO 3009027 --- [ocp-schedules-20,7b9b70dec84edab5,f2e0ab87adbf3a2c] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=79, maxSyncedRecordId=766198483195916288 2025-04-08 20:31:23.094 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:23.094 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:23.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:31:23.177985+08:00, lastProcessTime=2025-04-08T20:31:13.177017+08:00 2025-04-08 20:31:23.267 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:31:23.177985+08:00 2025-04-08 20:31:23.732 INFO 3009027 --- [ocp-schedules-23,d7dc8ad32faca2e2,5fcafd330030fa34] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:31:23.745 INFO 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:31:23.745 INFO 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:31:23.749 INFO 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:31:23.750 WARN 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:31:23.750 INFO 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:31:23.751 WARN 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:31:23.755 INFO 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:31:23.757 WARN 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:31:23.757 ERROR 3009027 --- [ocp-schedules-18,1d2e3cbabe66e92b,dcae0f7126d61017] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:31:24.292 INFO 3009027 --- [ocp-schedules-19,ea7cb81b935cc704,afe5cdd26f87a5fc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:31:24.336 INFO 3009027 --- [ocp-schedules-19,ea7cb81b935cc704,afe5cdd26f87a5fc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:31:24.336 INFO 3009027 --- [ocp-schedules-19,ea7cb81b935cc704,afe5cdd26f87a5fc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:31:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,9a56b44d7ed4fd0f,1c99292a8d7d3630] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:24.440 INFO 3009027 --- [ocp-schedules-28,af62ca7c5d2655fd,1fa47db800962374] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:31:24.946 WARN 3009027 --- [ocp-async-5,c51f787fd528257a,93b14a08d854f39c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:24.946 INFO 3009027 --- [ocp-async-4,5438fff1a389394e,a8d7258dd93dc653] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:25.961 INFO 3009027 --- [ocp-schedules-9,ef3012a076384947,9e8ce0623f77a13f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:31:25.962 INFO 3009027 --- [ocp-schedules-9,ef3012a076384947,9e8ce0623f77a13f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:31:26.094 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:26.095 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:26.947 WARN 3009027 --- [ocp-async-9,fb2d12561662dc4b,aa7cc91f250bc032] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:26.947 INFO 3009027 --- [ocp-async-4,7d07ec88dddb13d8,87ff1733d97e7754] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:31:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,e8003ca2c18bef9b,40044fcdc81a1343] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:28.947 WARN 3009027 --- [ocp-async-2,5cefa496f579964d,d1766406a1e2fa74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:28.947 INFO 3009027 --- [ocp-async-8,581ef77b9d33887a,fd4568b83413e565] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:29.095 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:29.096 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,813ee474a0ed585c,453bd1ad620ef923] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:30.947 WARN 3009027 --- [ocp-async-8,9cf0d2df20f95d14,6fd5711234d6f3fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:30.947 INFO 3009027 --- [ocp-async-5,35c14c628ec92f55,f41805d45b10abea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:32.096 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:32.096 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:32.897 INFO 3009027 --- [ocp-schedules-3,79ecd1367049a54a,71847fa55ed7d99f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:31:32.898 INFO 3009027 --- [ocp-schedules-14,a853812393122d57,5d5bcf4d135bb8bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:31:32.922 INFO 3009027 --- [ocp-async-9,3f6511cd0b1c617d,889af4e8f5294a54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:31:32.931 INFO 3009027 --- [ocp-schedules-5,a5fef0afeb9dac82,133559e3d4d97869] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:31:32.932 INFO 3009027 --- [ocp-schedules-2,3d92593558270798,f01654cad8ff5aa7] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:31:32.932 INFO 3009027 --- [ocp-schedules-2,3d92593558270798,f01654cad8ff5aa7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:32.933 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:31:32.947 WARN 3009027 --- [ocp-async-2,b3bb2d4b44549ed6,c2d20fcdb4bbf34e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:32.947 INFO 3009027 --- [ocp-async-3,55ce86236869640b,65d9accdb44c41ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:31:33.177002+08:00, lastProcessTime=2025-04-08T20:31:23.177985+08:00 2025-04-08 20:31:33.311 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:31:33.177002+08:00 2025-04-08 20:31:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,634698c76c4ec310,f9a20c6bd835a46d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:34.475 INFO 3009027 --- [ocp-schedules-25,6a43a2373ea407d3,ac6a991d294107a5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:31:34.947 WARN 3009027 --- [ocp-async-3,dada0a3c30586cd1,b816ee8d2fce5c13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:34.947 INFO 3009027 --- [ocp-async-8,b8ed666d9907b0bf,fd185112201877ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:35.096 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:35.097 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:35.969 INFO 3009027 --- [ocp-schedules-28,605df5ccb172248a,b0b8d107a7c09f46] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:31:35.971 INFO 3009027 --- [ocp-schedules-28,605df5ccb172248a,b0b8d107a7c09f46] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:31:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,80f766d68380d307,8ac0b62adfc0323c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:36.947 WARN 3009027 --- [ocp-async-6,d7a33c4921f06abf,87f2652bb199d5c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:36.947 INFO 3009027 --- [ocp-async-7,4da406a6c22170ad,42f02ace6b9bcb2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:38.097 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:38.098 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:38.947 WARN 3009027 --- [ocp-async-4,fbe62ceae2feaccc,affd988b286a5355] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:38.947 INFO 3009027 --- [ocp-async-9,36475f4cf9e8fd01,6b4f0ba99fa0fb30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,6458642d10e50c9e,d9645015f9767c46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:40.001 INFO 3009027 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:31:40.001 INFO 3009027 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:31:40.948 WARN 3009027 --- [ocp-async-10,e3d94e6afbec0a4c,3452930c66965b7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:40.948 INFO 3009027 --- [ocp-async-3,461582a360a58bb7,3d8e0c31d01bf413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:41.098 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:41.098 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,a81df1477854b390,02a9a524a802a689] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:42.922 INFO 3009027 --- [ocp-async-8,1e210d62f812343b,b53751b2f637bf14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:31:42.933 INFO 3009027 --- [ocp-schedules-21,dfea504c324d5b4f,f870f477c56fc1f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:31:42.933 INFO 3009027 --- [ocp-schedules-2,3d92593558270798,f01654cad8ff5aa7] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:31:42.948 WARN 3009027 --- [ocp-async-5,57e795b84253837a,a722058dcd9a09b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:42.948 INFO 3009027 --- [ocp-async-6,39601aeb68340be5,55a37a4447a69666] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:31:43.177130+08:00, lastProcessTime=2025-04-08T20:31:33.177002+08:00 2025-04-08 20:31:43.258 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:31:43.177130+08:00 2025-04-08 20:31:44.099 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:44.099 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:44.337 INFO 3009027 --- [ocp-schedules-27,d004f7995de43f45,a7939d41dc07e4ef] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:31:44.344 INFO 3009027 --- [ocp-schedules-27,d004f7995de43f45,a7939d41dc07e4ef] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:31:44.344 INFO 3009027 --- [ocp-schedules-27,d004f7995de43f45,a7939d41dc07e4ef] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:31:44.500 INFO 3009027 --- [ocp-schedules-5,3e50bb5624bc6439,add81e253cf30f8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:31:44.948 WARN 3009027 --- [ocp-async-4,9647a9342a3bdd6c,63cd19a687fad77d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:44.948 INFO 3009027 --- [ocp-async-9,c50025c2e790c0e1,5e5952381028e658] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,2c2ab59c66e016db,d5e013717064b8cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:45.982 INFO 3009027 --- [ocp-schedules-23,fe7358f9a7ba5a3d,bda8da109e191d27] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:31:45.984 INFO 3009027 --- [ocp-schedules-23,fe7358f9a7ba5a3d,bda8da109e191d27] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:31:46.948 WARN 3009027 --- [ocp-async-1,e337d671937ee854,b13335aa33c72919] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:46.948 INFO 3009027 --- [ocp-async-3,b5572249716e0881,5557f92052062aeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:47.100 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:47.100 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,2f42cae560a99fc0,52b3853da61dca98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:48.948 WARN 3009027 --- [ocp-async-7,bfddb9e72c4713c6,8839b9767fef6bda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:48.948 INFO 3009027 --- [ocp-async-5,60ef436b55665cb8,576f64bc566a229d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:50.100 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:50.101 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:50.948 WARN 3009027 --- [ocp-async-6,2a9ef7a77f81c32b,6673acf7e865332f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:50.948 INFO 3009027 --- [ocp-async-2,70dbb6eda9f2e198,bf54fedb3b248a0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,eacc2f2fac0a99e9,49c47dfd40d58125] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:52.897 INFO 3009027 --- [ocp-schedules-11,c27fad0988f7fb8c,17f37109e0cee214] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:31:52.897 INFO 3009027 --- [ocp-schedules-28,80ea9438bd4c7f87,34bd8ff89265406b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:31:52.897 INFO 3009027 --- [ocp-schedules-30,a5d382a255c2b583,70c3d3b92a4a4155] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:31:52.898 INFO 3009027 --- [ocp-schedules-30,a5d382a255c2b583,70c3d3b92a4a4155] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:52.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:31:52.898 INFO 3009027 --- [ocp-schedules-11,c27fad0988f7fb8c,17f37109e0cee214] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:52.898 INFO 3009027 --- [ocp-schedules-18,476dd08bdbd9d8eb,81b88b2bf15586e5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:31:52.898 INFO 3009027 --- [ocp-schedules-18,476dd08bdbd9d8eb,81b88b2bf15586e5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:31:52.899 INFO 3009027 --- [ocp-schedules-7,1e1ed689a9113535,c2e5adf6bed7caad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:31:52.899 INFO 3009027 --- [ocp-schedules-7,1e1ed689a9113535,c2e5adf6bed7caad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:31:52.899 INFO 3009027 --- [ocp-schedules-8,dc31db8070caf61d,bc55ea5d2f5fdd17] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:31:52.899 INFO 3009027 --- [ocp-schedules-8,dc31db8070caf61d,bc55ea5d2f5fdd17] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:31:52.921 INFO 3009027 --- [ocp-schedules-3,05261700296c0060,87cb5aa8f79d4cc6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:31:52.922 INFO 3009027 --- [ocp-async-10,9279b71fd8da4853,6d5eacff6e746660] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:31:52.934 INFO 3009027 --- [ocp-schedules-25,de5d479ef774f25b,5c6eff06f70c4836] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:31:52.947 INFO 3009027 --- [ocp-schedules-16,05ae3e0baac0aaa2,f4953da879d19dc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:31:52.948 INFO 3009027 --- [ocp-schedules-16,6ed3992ac49f771e,bb85cc2a5a66d568] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:31:52.949 WARN 3009027 --- [ocp-async-8,7e03ab9b93c58d10,7a7c98cbe820394f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:52.949 INFO 3009027 --- [ocp-async-9,2fbdc0a5126b1f54,d594e465821fa47e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:52.949 INFO 3009027 --- [ocp-schedules-16,6ed3992ac49f771e,bb85cc2a5a66d568] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:31:52.955 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:31:52.955 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:31:52.961 INFO 3009027 --- [ocp-schedules-20,41c4a92cb0d85110,022aa0fc3494da8a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:31:53.101 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:53.102 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:31:53.178087+08:00, lastProcessTime=2025-04-08T20:31:43.177130+08:00 2025-04-08 20:31:53.272 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:31:53.178087+08:00 2025-04-08 20:31:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,7a5347dff04b72b0,f15a56d0cc5f89c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:54.554 INFO 3009027 --- [ocp-schedules-6,3911a5ce291aabff,5bc038f726c53c97] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:31:54.949 WARN 3009027 --- [ocp-async-9,e456f6963b135d62,ffe42c7c235a70e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:54.949 INFO 3009027 --- [ocp-async-3,031aecaa1b90b6ce,92cd44e9d8be9d50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:55.987 INFO 3009027 --- [ocp-schedules-6,640eeb557174386c,74d335dc6997101f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:31:55.989 INFO 3009027 --- [ocp-schedules-6,640eeb557174386c,74d335dc6997101f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:31:56.102 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:56.102 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:31:56.949 WARN 3009027 --- [ocp-async-2,967cf434136c9008,3d61c18d5f9e8daf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:56.949 INFO 3009027 --- [ocp-async-3,e9896c51084c3555,2d76bb23a0cc4347] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,9f2620d8e4c3db50,a669fc6f67c7346e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115520, interval=60 2025-04-08 20:31:58.949 INFO 3009027 --- [ocp-async-8,b7658dd387b941a0,f92cad57c2e1fa40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:31:58.949 WARN 3009027 --- [ocp-async-10,031503e1239d4f99,9860001c938e856a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:31:59.102 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:31:59.103 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,e64ef9b752c52907,fc0b29d359715119] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:00.949 WARN 3009027 --- [ocp-async-7,27aa6bb1543177bd,f1aad14ffa8b1342] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:00.949 INFO 3009027 --- [ocp-async-9,929271226cf15ab3,f21bdbbd332844b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:02.103 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:02.104 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:02.898 INFO 3009027 --- [ocp-schedules-11,c27fad0988f7fb8c,17f37109e0cee214] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:32:02.898 INFO 3009027 --- [ocp-schedules-30,a5d382a255c2b583,70c3d3b92a4a4155] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:32:02.922 INFO 3009027 --- [ocp-async-9,5bb7b0506f486728,223891b1bf45be63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:32:02.935 INFO 3009027 --- [ocp-schedules-26,ef9dda4570d8c2a7,664e0e252c0826dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:32:02.949 INFO 3009027 --- [ocp-async-4,428cb9afdabeb587,f1700ca350c5948a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:02.949 WARN 3009027 --- [ocp-async-3,cf2e6facc7922efb,e422791da1668569] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:32:03.176926+08:00, lastProcessTime=2025-04-08T20:31:53.178087+08:00 2025-04-08 20:32:03.295 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:32:03.176926+08:00 2025-04-08 20:32:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,bdd23d20b2b819bd,9d0423950b33ec98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:04.346 INFO 3009027 --- [ocp-schedules-24,4f356aaca9892652,9ccd00496be4a767] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:32:04.359 INFO 3009027 --- [ocp-schedules-24,4f356aaca9892652,9ccd00496be4a767] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:32:04.359 INFO 3009027 --- [ocp-schedules-24,4f356aaca9892652,9ccd00496be4a767] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:32:04.597 INFO 3009027 --- [ocp-schedules-10,86b4c95f1dba051f,e43d3c92791f5946] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:32:04.950 WARN 3009027 --- [ocp-async-1,25e8d658163d9252,dae510b483dbbb65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:04.950 INFO 3009027 --- [ocp-async-4,daea0da1464eba6a,db905dbd7aa11f13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:05.104 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:05.105 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:06.040 INFO 3009027 --- [ocp-schedules-2,1f050b153263df82,f0207e12b71fe6c5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:32:06.041 INFO 3009027 --- [ocp-schedules-2,1f050b153263df82,f0207e12b71fe6c5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:32:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,d250777ea7b7380b,b663b7f837e9de00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:06.950 WARN 3009027 --- [ocp-async-5,94bd9e8dd5318db0,77ba2e5d51a48e28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:06.950 INFO 3009027 --- [ocp-async-6,46a78a3729ea8e45,8e47e81d101e0a73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:08.105 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:08.105 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:08.950 WARN 3009027 --- [ocp-async-6,5db1d138c141bf13,f23a18909f53b057] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:08.950 INFO 3009027 --- [ocp-async-3,d4bca18494cf29ee,f06f77ab4b661aac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,5aa167e9d575b66e,7e8dabb3055bece0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:10.002 INFO 3009027 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:32:10.002 INFO 3009027 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:32:10.950 WARN 3009027 --- [ocp-async-3,016151de6c13188f,05513b5ff3373f6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:10.950 INFO 3009027 --- [ocp-async-10,5a1caa66d345a1e5,b5f9dd0113868134] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:10.990 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:32:11.106 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:11.106 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,50406ba997d4aafb,c99d213bb9a6dc79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:12.408 INFO 3009027 --- [prometheus-cache-monitor1,27522820ed81d36c,621720ca7598aa6e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1622,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":417,"secondWriteQueue":0,"minuteWriteCacheSize":414} 2025-04-08 20:32:12.408 INFO 3009027 --- [prometheus-cache-monitor1,27522820ed81d36c,621720ca7598aa6e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=61, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=61, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:32:12.922 INFO 3009027 --- [ocp-async-10,3cbb9ef6743eae5c,4e6d41c1de35e608] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:32:12.936 INFO 3009027 --- [ocp-schedules-22,c312ca28f67ccebe,a1e91038401659df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:32:12.950 WARN 3009027 --- [ocp-async-5,1b63a8feba131b42,13cd88abbe7cdeac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:12.950 INFO 3009027 --- [ocp-async-7,50376b4eca8c4f37,1f6668cf3614437f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:32:13.176474+08:00, lastProcessTime=2025-04-08T20:32:03.176926+08:00 2025-04-08 20:32:13.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:32:13.176474+08:00 2025-04-08 20:32:14.106 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:14.107 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:14.634 INFO 3009027 --- [ocp-schedules-24,440d598fb0f887ae,0fa6b716513e29e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:32:14.950 WARN 3009027 --- [ocp-async-9,f1308a31782cac31,831a5ecff8432d1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:14.950 INFO 3009027 --- [ocp-async-2,26a7f03ad9a9a206,6ad71c91ecfa9b1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,2d6eff6d18087e53,35872c4d4c42d13b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115460, nodeCount=12, affectRows=12 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115460, nodeCount=12, validCount=12 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115460, nodeCount=11, affectRows=11 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115460, nodeCount=359, affectRows=359 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115460, nodeCount=11, affectRows=11 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115460, nodeCount=14, affectRows=14 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115460, nodeCount=9, affectRows=9 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115460, nodeCount=11, validCount=11 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115460, nodeCount=11, validCount=11 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115460, nodeCount=8, validCount=8 2025-04-08 20:32:15.540 INFO 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115460, nodeCount=14, validCount=14 2025-04-08 20:32:15.542 INFO 3009027 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115460, nodeCount=357, validCount=357 2025-04-08 20:32:16.068 INFO 3009027 --- [ocp-schedules-29,a473866990ec72e3,c15426a78d794509] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:32:16.070 INFO 3009027 --- [ocp-schedules-29,a473866990ec72e3,c15426a78d794509] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:32:16.950 WARN 3009027 --- [ocp-async-8,f64da743d09190a6,6d7cb3f97deec191] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:16.950 INFO 3009027 --- [ocp-async-2,023121692ea0d7cb,beafbd60c7249ef8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:17.107 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:17.108 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,96c1e21914bccb94,f3f85a50b3a6a256] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:18.951 WARN 3009027 --- [ocp-async-6,573509301f436f69,a40a0742408c95bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:18.951 INFO 3009027 --- [ocp-async-2,7e6be364c91dea18,7f376255dbf83a0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:20.108 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:20.108 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:20.951 WARN 3009027 --- [ocp-async-9,7a3c22230d9d092d,b08bf0ed6e402606] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:20.951 INFO 3009027 --- [ocp-async-7,6e56023569538dd3,751897decdd98c5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,5dbe6a34011c03ce,fcf14a7e1e4fd481] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:22.895 INFO 3009027 --- [ocp-schedules-19,4296b7c57c288ac4,370a215767e2ffca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:32:22.896 INFO 3009027 --- [ocp-schedules-8,f66d56fc1995ac33,5f2f0a50960c4ac3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:32:22.896 INFO 3009027 --- [ocp-schedules-5,72462ea49c3571ca,da8a433b2934684b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:32:22.896 INFO 3009027 --- [ocp-schedules-11,0d68a1554fc86073,546775c136898567] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:32:22.896 INFO 3009027 --- [ocp-schedules-29,c25957d37adc0a24,ca8257f5fc26ad86] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:32:22.897 INFO 3009027 --- [ocp-schedules-5,72462ea49c3571ca,da8a433b2934684b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:22.897 INFO 3009027 --- [ocp-schedules-29,c25957d37adc0a24,ca8257f5fc26ad86] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:32:22.898 INFO 3009027 --- [ocp-schedules-1,c13bf433f160c0a1,59a7e1396c6803a3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:32:22.898 INFO 3009027 --- [ocp-schedules-1,c13bf433f160c0a1,59a7e1396c6803a3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:32:22.898 INFO 3009027 --- [ocp-schedules-7,a918dc2cd4d331c1,bc4da917da14003b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:32:22.898 INFO 3009027 --- [ocp-schedules-7,a918dc2cd4d331c1,bc4da917da14003b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:32:22.898 INFO 3009027 --- [ocp-schedules-9,faff2095dd76c4a6,c11107f4763c0da2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:32:22.898 INFO 3009027 --- [ocp-schedules-9,faff2095dd76c4a6,c11107f4763c0da2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:32:22.902 INFO 3009027 --- [ocp-schedules-11,0d68a1554fc86073,546775c136898567] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:32:22.913 INFO 3009027 --- [ocp-schedules-28,affc4e7b5d921c7d,7216bb17bfc5a4b3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1652,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":414,"seriesIdKeySize":417,"secondWriteQueue":0,"minuteWriteCacheSize":414} 2025-04-08 20:32:22.913 INFO 3009027 --- [ocp-schedules-28,affc4e7b5d921c7d,7216bb17bfc5a4b3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=61, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=61, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:32:22.922 INFO 3009027 --- [ocp-schedules-24,77d9c5f25f1a6af5,c20ba2f0aeb2fb07] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:32:22.922+08:00, clusters=[1] 2025-04-08 20:32:22.922 INFO 3009027 --- [ocp-async-1,48667a366095b77e,cb9272544b681f85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:32:22.923 INFO 3009027 --- [ocp-schedules-24,77d9c5f25f1a6af5,c20ba2f0aeb2fb07] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:32:22.927 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:32:22.927 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:32:22.931 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:32:22.931 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:32:22.931 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:32:22.933 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:32:22.933 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:32:22.933 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:32:22.935 INFO 3009027 --- [ocp-schedules-2,ca9d975a8bd1b20b,f5e0c36f3306f3b9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:32:22.935+08:00, clusters=[1] 2025-04-08 20:32:22.935 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:32:22.935 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:32:22.935 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:32:22.936 INFO 3009027 --- [ocp-schedules-2,ca9d975a8bd1b20b,f5e0c36f3306f3b9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:32:22.937 INFO 3009027 --- [ocp-schedules-15,a8960b0c04cd1c49,a21537651bc4a67b] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:32:22.937 INFO 3009027 --- [ocp-schedules-15,42f4097a42682ac1,59a7954bea6667fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:32:22.944 INFO 3009027 --- [ocp-schedules-13,68503477f649cfc7,12614b3bbae74e17] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:32:22.944 INFO 3009027 --- [ocp-schedules-19,4296b7c57c288ac4,370a215767e2ffca] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:32:22.945 INFO 3009027 --- [ocp-schedules-30,f6f3dcce1e237929,f67bdd3f7ef9cd35] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:32:22.946 INFO 3009027 --- [ocp-schedules-30,f6f3dcce1e237929,f67bdd3f7ef9cd35] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:22.950 INFO 3009027 --- [ocp-schedules-10,37f5ce1dd1e0f475,34179d7827fb3be6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:32:22.951 WARN 3009027 --- [ocp-async-6,f1796b0957ed56b3,0463f74840bfd329] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:22.951 INFO 3009027 --- [ocp-async-4,6653538f285cea7e,56245b63fbd43a9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:22.955 INFO 3009027 --- [ocp-schedules-16,6ed3992ac49f771e,bb85cc2a5a66d568] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:32:22.955 INFO 3009027 --- [ocp-schedules-16,8416460e58309343,9d28155c566c1781] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:32:22.956 INFO 3009027 --- [ocp-schedules-16,8416460e58309343,9d28155c566c1781] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:22.957 ERROR 3009027 --- [ocp-schedules-30,f6f3dcce1e237929,f67bdd3f7ef9cd35] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:32:22.957 INFO 3009027 --- [ocp-schedules-30,f6f3dcce1e237929,f67bdd3f7ef9cd35] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:32:22.958 INFO 3009027 --- [ocp-schedules-1,ca2602eed4bb6d53,1c09bc76998c3980] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:32:22.959 INFO 3009027 --- [ocp-schedules-1,ca2602eed4bb6d53,1c09bc76998c3980] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:22.961 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:32:22.961 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:32:22.962 INFO 3009027 --- [ocp-schedules-7,64ef32e8c3d48b7b,4175c63f592c81cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:32:22.966 INFO 3009027 --- [ocp-schedules-28,dd4d49c1f7e7fe04,0aaf70a332d02631] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:32:22.966 INFO 3009027 --- [ocp-schedules-28,dd4d49c1f7e7fe04,0aaf70a332d02631] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:22.970 INFO 3009027 --- [ocp-schedules-4,2dd398ff314e19e7,4aece4da88cd7c70] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:32:22.970 INFO 3009027 --- [ocp-schedules-4,2dd398ff314e19e7,4aece4da88cd7c70] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:32:22.971 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:32:22.971 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:32:22.974 INFO 3009027 --- [ocp-schedules-4,2dd398ff314e19e7,4aece4da88cd7c70] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:32:22.974 INFO 3009027 --- [ocp-schedules-4,2dd398ff314e19e7,4aece4da88cd7c70] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:32:22.974 INFO 3009027 --- [ocp-schedules-25,647e1342b2eca14f,63cfa4ea0e4c8356] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:32:22.974 INFO 3009027 --- [ocp-schedules-25,647e1342b2eca14f,63cfa4ea0e4c8356] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:32:22.982 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:32:22.983 ERROR 3009027 --- [ocp-schedules-28,dd4d49c1f7e7fe04,0aaf70a332d02631] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:32:22.983 INFO 3009027 --- [ocp-schedules-28,dd4d49c1f7e7fe04,0aaf70a332d02631] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:32:22.995 INFO 3009027 --- [ocp-schedules-14,3e1f40d7752deace,59e9871d0ced94fa] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:32:22.995 INFO 3009027 --- [ocp-schedules-14,3e1f40d7752deace,59e9871d0ced94fa] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:32:22.998 INFO 3009027 --- [ocp-schedules-24,25ffed108fc0a380,15ec1f8c467f3360] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:32:22.999 INFO 3009027 --- [ocp-schedules-24,25ffed108fc0a380,15ec1f8c467f3360] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:32:22.999 INFO 3009027 --- [ocp-schedules-24,25ffed108fc0a380,15ec1f8c467f3360] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:32:23.051 INFO 3009027 --- [ocp-schedules-27,2198044a7989894d,d5c7c43abdda4ab9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:32:23.086 INFO 3009027 --- [ocp-schedules-18,9d2f147fcab77bda,5615e489344deeac] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:32:23.086 INFO 3009027 --- [ocp-schedules-18,9d2f147fcab77bda,5615e489344deeac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:32:23.087 INFO 3009027 --- [ocp-schedules-18,9d2f147fcab77bda,5615e489344deeac] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:32:23.087 INFO 3009027 --- [ocp-schedules-18,9d2f147fcab77bda,5615e489344deeac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198297748959232 2025-04-08 20:32:23.087 INFO 3009027 --- [ocp-schedules-18,9d2f147fcab77bda,5615e489344deeac] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:32:23.087 INFO 3009027 --- [ocp-schedules-18,9d2f147fcab77bda,5615e489344deeac] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:32:23.109 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:23.109 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:32:23.176636+08:00, lastProcessTime=2025-04-08T20:32:13.176474+08:00 2025-04-08 20:32:23.259 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:32:23.176636+08:00 2025-04-08 20:32:23.751 INFO 3009027 --- [ocp-schedules-12,6b355c65f345fa46,709544cabdb1d568] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:32:23.761 INFO 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:32:23.761 INFO 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:32:23.764 INFO 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:32:23.765 WARN 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:32:23.765 INFO 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:32:23.767 WARN 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:32:23.770 INFO 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:32:23.771 WARN 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:32:23.772 ERROR 3009027 --- [ocp-schedules-9,cb9a7c56a8eb309f,b2bbe09f9f59e7c9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:32:24.361 INFO 3009027 --- [ocp-schedules-14,192e542dfa6bf5e7,4f3c269e5a96f1e8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:32:24.372 INFO 3009027 --- [ocp-schedules-14,192e542dfa6bf5e7,4f3c269e5a96f1e8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:32:24.372 INFO 3009027 --- [ocp-schedules-14,192e542dfa6bf5e7,4f3c269e5a96f1e8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:32:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,be6fe1f58df85acf,7b85f56a229c6dc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:24.658 INFO 3009027 --- [ocp-schedules-22,2f871ab0bf904b13,8e221c393115b2c9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:32:24.951 WARN 3009027 --- [ocp-async-4,fe33c3e7050abfc1,e992b70aa43d839a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:24.951 INFO 3009027 --- [ocp-async-1,8494e030c5275008,576a72c99d100898] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:26.073 INFO 3009027 --- [ocp-schedules-18,34bf92b6ec41fd5b,5ce5982ffa0a7f16] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:32:26.074 INFO 3009027 --- [ocp-schedules-18,34bf92b6ec41fd5b,5ce5982ffa0a7f16] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:32:26.109 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:26.110 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:26.951 WARN 3009027 --- [ocp-async-2,f7a21a3f4a832a31,55b93bb28009acc5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:26.951 INFO 3009027 --- [ocp-async-8,4ead0d220086365a,784d8fa27cbd93d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:32:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,032a3639e86e7dff,9bead7e34c6a3049] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:28.951 WARN 3009027 --- [ocp-async-9,4f0f469458c42b2f,a2fc166eaad001b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:28.951 INFO 3009027 --- [ocp-async-6,f33ea5a2390ae6b7,0dd84b991206ed20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:29.110 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:29.111 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,39d7f41060483f58,826a9edb88503268] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:30.951 WARN 3009027 --- [ocp-async-5,2b6754069d41cf46,eb6b6e8fb7cd2046] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:30.951 INFO 3009027 --- [ocp-async-3,7b2d5baa4db43b79,6fc14d6c6b5d554f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:32.111 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:32.112 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:32.897 INFO 3009027 --- [ocp-schedules-5,72462ea49c3571ca,da8a433b2934684b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:32:32.898 INFO 3009027 --- [ocp-schedules-29,c25957d37adc0a24,ca8257f5fc26ad86] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:32:32.923 INFO 3009027 --- [ocp-async-2,ab79fb6e9ffccced,9094d1c38d039889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:32:32.938 INFO 3009027 --- [ocp-schedules-3,bb92a1af7520f9cc,a87eee76475c7e69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:32:32.952 INFO 3009027 --- [ocp-async-8,f064a27177e27f00,c70d9717ee0305b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:32.952 WARN 3009027 --- [ocp-async-9,90c561c6a12cce1a,5d2d8d45db8e6a9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:33.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:32:33.177766+08:00, lastProcessTime=2025-04-08T20:32:23.176636+08:00 2025-04-08 20:32:33.283 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:32:33.177766+08:00 2025-04-08 20:32:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,4f943f2c03391616,0d6190ca8d7048a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:34.688 INFO 3009027 --- [ocp-schedules-8,68b6cec61f100f6e,3240033ef50fc653] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:32:34.952 WARN 3009027 --- [ocp-async-2,f538c1bc05593c3f,7dc29f3879d2593f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:34.952 INFO 3009027 --- [ocp-async-5,2d2a8f8b2bf8e005,c0a2222e86888e75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:35.112 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:35.112 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:36.194 INFO 3009027 --- [ocp-schedules-3,9ba5a28ec990653f,6cfa724edb2ddb33] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:32:36.195 INFO 3009027 --- [ocp-schedules-3,9ba5a28ec990653f,6cfa724edb2ddb33] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:32:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,8703a6516771578e,8c4b5745be5ff394] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:36.952 WARN 3009027 --- [ocp-async-4,5faa431bba625926,55aa94c5931b161e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:36.952 INFO 3009027 --- [ocp-async-3,aafc3ef0de59ecf7,c180831266a61a1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:38.112 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:38.113 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:38.952 WARN 3009027 --- [ocp-async-10,1dad9c08a194b25f,0f94e1c42205f20c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:38.952 INFO 3009027 --- [ocp-async-7,e0cc6402ffae32be,95cabd7898093938] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,44034f55edaf8fb2,558a6f87f0bd476c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:40.002 INFO 3009027 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:32:40.002 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:32:40.952 WARN 3009027 --- [ocp-async-8,bba5cd2aa5645908,dba7828e56c2b4e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:40.952 INFO 3009027 --- [ocp-async-6,3b73635fd8e73a59,aabca64b89da0665] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:40.953 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:32:40.957 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1), status=TAKINGOVER 2025-04-08 20:32:40.957 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 20:32:41.113 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:41.114 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,b8146e29b4b25edf,28e45cf90997c7b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:42.923 INFO 3009027 --- [ocp-async-5,4480634452c543c3,a800481fd9df5878] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:32:42.933 INFO 3009027 --- [ocp-schedules-20,b0959c5b8509d8d3,012191093fc438aa] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:32:42.934 INFO 3009027 --- [ocp-schedules-20,b0959c5b8509d8d3,012191093fc438aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:42.934 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:32:42.939 INFO 3009027 --- [ocp-schedules-11,1a562da44058168c,90a9019a4868500d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:32:42.952 WARN 3009027 --- [ocp-async-1,f4d0da3881c9181e,a7a399fcc3091b27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:42.952 INFO 3009027 --- [ocp-async-3,16c510fb7e2a7aaf,609247b628f493c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:32:43.176850+08:00, lastProcessTime=2025-04-08T20:32:33.177766+08:00 2025-04-08 20:32:43.263 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:32:43.176850+08:00 2025-04-08 20:32:44.114 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:44.115 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:44.373 INFO 3009027 --- [ocp-schedules-17,8b236b8d31a39ca0,267ef824a4ada912] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:32:44.444 INFO 3009027 --- [ocp-schedules-17,8b236b8d31a39ca0,267ef824a4ada912] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:32:44.444 INFO 3009027 --- [ocp-schedules-17,8b236b8d31a39ca0,267ef824a4ada912] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:32:44.712 INFO 3009027 --- [ocp-schedules-7,0e197e1ae3da8130,2ec6ca3e12007a56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:32:44.952 WARN 3009027 --- [ocp-async-10,f043e20e27956978,d2266dcf9e709908] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:44.953 INFO 3009027 --- [ocp-async-7,cf8829e2f1562a86,6cba9806038cef80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,6fa0b839a8e33333,6c03c6b54a696c35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:46.202 INFO 3009027 --- [ocp-schedules-22,8e4d64b3bedf19fc,d98060fd198ec4fb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:32:46.203 INFO 3009027 --- [ocp-schedules-22,8e4d64b3bedf19fc,d98060fd198ec4fb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:32:46.953 WARN 3009027 --- [ocp-async-6,d5cea21c857b2bf7,132161e5a82afbf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:46.953 INFO 3009027 --- [ocp-async-2,c0ca7fefae7603f4,31562538ce1c008f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:47.115 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:47.115 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,eefd5b763897adfe,95bb72afa71a1d7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:48.953 WARN 3009027 --- [ocp-async-1,0feff8242f256565,27fb9d6a7a65ff95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:48.953 INFO 3009027 --- [ocp-async-5,67c6becd6556e740,a79b117026617824] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:50.115 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:50.116 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:50.953 WARN 3009027 --- [ocp-async-9,9144924c9ce68bcc,3d240f2008808a9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:50.953 INFO 3009027 --- [ocp-async-10,ae65be021a51ee3f,804dfe63c02ab195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,0a2a1d4c6b813eac,a8f913e44c9dcd4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:52.896 INFO 3009027 --- [ocp-schedules-24,e0db1e4d537569ba,baf653d8b2739af5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:32:52.896 INFO 3009027 --- [ocp-schedules-7,ab6ff6851951e1e0,7ad63c0c7d5afde9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:32:52.896 INFO 3009027 --- [ocp-schedules-26,6f25f33c9224602d,0abd1c3feb14f875] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:32:52.897 INFO 3009027 --- [ocp-schedules-7,ab6ff6851951e1e0,7ad63c0c7d5afde9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:52.897 INFO 3009027 --- [ocp-schedules-26,6f25f33c9224602d,0abd1c3feb14f875] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:32:52.898 INFO 3009027 --- [ocp-schedules-27,ddd19b9d7db07cfb,ac53ff24b8548fc6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:32:52.898 INFO 3009027 --- [ocp-schedules-19,12aa411a27d100e4,79911db302161d2d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:32:52.898 INFO 3009027 --- [ocp-schedules-27,ddd19b9d7db07cfb,ac53ff24b8548fc6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:32:52.898 INFO 3009027 --- [ocp-schedules-19,12aa411a27d100e4,79911db302161d2d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:32:52.899 INFO 3009027 --- [ocp-schedules-10,829c6d7a0c94351a,ed252344b7a1f0d2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:32:52.899 INFO 3009027 --- [ocp-schedules-10,829c6d7a0c94351a,ed252344b7a1f0d2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:32:52.917 INFO 3009027 --- [ocp-schedules-9,2f2a6aadc001c0fc,16d34cd2104cbf7f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:32:52.923 INFO 3009027 --- [ocp-async-10,6f0de23fecdd4e25,5abcc2add20de429] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:32:52.934 INFO 3009027 --- [ocp-schedules-20,b0959c5b8509d8d3,012191093fc438aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:32:52.941 INFO 3009027 --- [ocp-schedules-4,828b00eb33bc0aca,942124192d332512] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:32:52.953 INFO 3009027 --- [ocp-async-6,c418d7255b5a01dd,f45278e0198545f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:52.953 WARN 3009027 --- [ocp-async-8,a7cc392b46730218,938757f13a773314] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:52.962 INFO 3009027 --- [ocp-schedules-16,8416460e58309343,9d28155c566c1781] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:32:52.962 INFO 3009027 --- [ocp-schedules-16,5925f2c639707423,3f842944cc240d45] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:32:52.963 INFO 3009027 --- [ocp-schedules-16,5925f2c639707423,3f842944cc240d45] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:32:52.963 INFO 3009027 --- [ocp-schedules-13,0cd662036197917c,ee9ff3b24eaa2bf5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:32:52.967 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:32:52.967 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:32:53.116 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:53.117 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:32:53.176838+08:00, lastProcessTime=2025-04-08T20:32:43.176850+08:00 2025-04-08 20:32:53.291 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:32:53.176838+08:00 2025-04-08 20:32:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,82a34a4716f69aa2,4cec2cc6871e46ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:54.744 INFO 3009027 --- [ocp-schedules-8,71699ebdaef56a73,b5efdf478820d367] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:32:54.953 INFO 3009027 --- [ocp-async-6,95a4f7297c9ad554,36263772af037308] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:54.953 WARN 3009027 --- [ocp-async-4,14f611f00e61e886,bcc296d2df881c9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:56.117 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:56.117 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:32:56.219 INFO 3009027 --- [ocp-schedules-11,d8d3672393ac5018,5a66897e365f6071] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:32:56.220 INFO 3009027 --- [ocp-schedules-11,d8d3672393ac5018,5a66897e365f6071] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:32:56.953 WARN 3009027 --- [ocp-async-3,dcbda869e8005a13,cac2b30c0bdea993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:56.953 INFO 3009027 --- [ocp-async-9,f557206b66c65812,e1137234bafe8721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,343478e0a9bc7ec7,b83a8f822fc8f890] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115580, interval=60 2025-04-08 20:32:58.954 WARN 3009027 --- [ocp-async-2,99fc7397828907aa,c24af0b022d93495] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:32:58.954 INFO 3009027 --- [ocp-async-9,3e692d018dfeec2e,d1f00227bba5bb98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:32:59.118 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:32:59.118 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,cbdfc3f37a14f7e2,f52f3875bbfe9e1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:00.954 WARN 3009027 --- [ocp-async-4,c47455280306d418,fec90775053479e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:00.954 INFO 3009027 --- [ocp-async-10,8149ace9b6d9dbab,670f66d1fb0edf05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:02.118 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:02.119 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:02.897 INFO 3009027 --- [ocp-schedules-7,ab6ff6851951e1e0,7ad63c0c7d5afde9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:33:02.898 INFO 3009027 --- [ocp-schedules-26,6f25f33c9224602d,0abd1c3feb14f875] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:33:02.923 INFO 3009027 --- [ocp-async-6,13f32f2b287e20fc,3632aadb391643f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:33:02.942 INFO 3009027 --- [ocp-schedules-17,5213252ac7032a59,e30adcb00b3320b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:33:02.954 WARN 3009027 --- [ocp-async-10,bb6014192946f2d1,734cb5e9e0669605] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:02.954 INFO 3009027 --- [ocp-async-7,b67cb5d0f202cd93,20a69e68d847010b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:33:03.176737+08:00, lastProcessTime=2025-04-08T20:32:53.176838+08:00 2025-04-08 20:33:03.297 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:33:03.176737+08:00 2025-04-08 20:33:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,4054176a87dee2b2,147ba329b020bef2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:04.446 INFO 3009027 --- [ocp-schedules-21,9d79eac92239293d,547c026940c97283] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:33:04.492 INFO 3009027 --- [ocp-schedules-21,9d79eac92239293d,547c026940c97283] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:33:04.492 INFO 3009027 --- [ocp-schedules-21,9d79eac92239293d,547c026940c97283] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:33:04.768 INFO 3009027 --- [ocp-schedules-3,6824868e85c5b57e,811680a3e216368d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:33:04.954 WARN 3009027 --- [ocp-async-2,973649a91314f81f,a75ef9ac8bf0a8b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:04.954 INFO 3009027 --- [ocp-async-1,94278702b317e34c,5f9f99374a4c305e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:05.119 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:05.120 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:06.290 INFO 3009027 --- [ocp-schedules-27,5d539aac83d4221e,92f860ae9e0c2d62] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:33:06.292 INFO 3009027 --- [ocp-schedules-27,5d539aac83d4221e,92f860ae9e0c2d62] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:33:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,7c9d5e3dbd7a6226,2362c09ff1954cdd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:06.954 WARN 3009027 --- [ocp-async-3,d0a75fb968d2f055,0f8106f1cf82c629] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:06.954 INFO 3009027 --- [ocp-async-5,5c8d0222675fbe20,2cc14af2e4963a80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:08.120 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:08.120 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:08.954 INFO 3009027 --- [ocp-async-5,08e1321f85449eb9,360a85b35bb2d48a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:08.954 WARN 3009027 --- [ocp-async-8,aa3e39ce627ac924,f6967eaf91c9a902] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,a525ce80d91847f9,c573747f5edc0366] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:10.001 INFO 3009027 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:33:10.001 INFO 3009027 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:33:10.954 WARN 3009027 --- [ocp-async-7,78c80be79410601c,abfe09b301672279] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:10.954 INFO 3009027 --- [ocp-async-2,b8724b6e84ebd2d7,dece68d857bc852f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:10.989 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:33:11.120 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:11.121 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,1ca6fb338af9e40a,b118f57df4071c80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:12.408 INFO 3009027 --- [prometheus-cache-monitor1,89cc35a0d6f78f1a,a6767ccba84ee6d2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1652,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":417,"secondWriteQueue":0,"minuteWriteCacheSize":414} 2025-04-08 20:33:12.408 INFO 3009027 --- [prometheus-cache-monitor1,89cc35a0d6f78f1a,a6767ccba84ee6d2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=61, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=61, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:33:12.889 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:29:12.756, end=2025-04-08T20:31:12.885} 2025-04-08 20:33:12.892 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:33:12.892 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:33:12.923 INFO 3009027 --- [ocp-async-2,3b8792cb59d8551a,94be07fe6cea9cf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:33:12.943 INFO 3009027 --- [ocp-schedules-27,51912a7fdfdd10c4,298116178f385a4b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:33:12.955 WARN 3009027 --- [ocp-async-4,38d44e914bfeefe5,7a47c15c5cbb0324] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:12.955 INFO 3009027 --- [ocp-async-1,603a8cca9ad5c2d4,fc21f916813bec36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:33:13.176367+08:00, lastProcessTime=2025-04-08T20:33:03.176737+08:00 2025-04-08 20:33:13.290 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:33:13.176367+08:00 2025-04-08 20:33:14.121 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:14.122 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:14.801 INFO 3009027 --- [ocp-schedules-14,5a6d78b255b625d2,f244ca78cc58f2b6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:33:14.955 WARN 3009027 --- [ocp-async-6,86f146c1ddf6a8bd,c3898342837a035c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:14.955 INFO 3009027 --- [ocp-async-8,ed0100e4148ea36d,eb708485a55a9e8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,62b27a9e67d4cd59,80b15cc0b72bdf66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:15.607 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115520, nodeCount=416, affectRows=416 2025-04-08 20:33:15.610 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115520, nodeCount=413, validCount=413 2025-04-08 20:33:16.303 INFO 3009027 --- [ocp-schedules-12,7d4f731f4c37c3f1,df89e63008a68f36] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:33:16.304 INFO 3009027 --- [ocp-schedules-12,7d4f731f4c37c3f1,df89e63008a68f36] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:33:16.955 WARN 3009027 --- [ocp-async-5,874487f8b39a4b68,cecd1f5bbb9c7214] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:16.955 INFO 3009027 --- [ocp-async-8,09e3ffae0213d112,4ad8e68b7d89dfc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:17.122 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:17.123 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,84631e2da6955be5,955f1fa326ea80f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:18.955 WARN 3009027 --- [ocp-async-4,81f6b8ea01892ffd,0a4fc632f4917c0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:18.955 INFO 3009027 --- [ocp-async-3,8622506da7a5b9ad,f6aa390aaa7bc924] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:20.123 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:20.123 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:20.955 WARN 3009027 --- [ocp-async-7,70cf9dc0dcaace4c,b6ad733b21844367] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:20.955 INFO 3009027 --- [ocp-async-6,28d81f369b386f9e,dab634e62d5dbdb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,174b08dfa5d2cd80,18fc39f05531bd0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:22.895 INFO 3009027 --- [ocp-schedules-28,59822f8beb57008f,80661d281c6ff0b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:33:22.896 INFO 3009027 --- [ocp-schedules-30,168a54dbb810fddf,2c2491f6cae28360] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:33:22.896 INFO 3009027 --- [ocp-schedules-8,7b3e02cf4faa3acd,0470e66eeb78e68f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:33:22.896 INFO 3009027 --- [ocp-schedules-7,0239fd52fbca7b24,4ef201db5df5a562] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:33:22.896 INFO 3009027 --- [ocp-schedules-8,7b3e02cf4faa3acd,0470e66eeb78e68f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:22.897 INFO 3009027 --- [ocp-schedules-6,06162e911139ca57,de67e6af9257851e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-6,06162e911139ca57,de67e6af9257851e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:22.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-13,b3239f399fab1de5,180af1d2d74291fe] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-11,33fbfb99a27788ac,d8f8681aad48d553] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-13,b3239f399fab1de5,180af1d2d74291fe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-11,33fbfb99a27788ac,d8f8681aad48d553] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-23,a4b50aff046f3fa7,119b95ad0a1631ba] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:33:22.898 INFO 3009027 --- [ocp-schedules-23,a4b50aff046f3fa7,119b95ad0a1631ba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:33:22.902 INFO 3009027 --- [ocp-schedules-7,0239fd52fbca7b24,4ef201db5df5a562] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:33:22.910 INFO 3009027 --- [ocp-schedules-28,59822f8beb57008f,80661d281c6ff0b1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:33:22.917 INFO 3009027 --- [ocp-schedules-21,cda010904ccd0656,01c1719b876353a8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:33:22.923 INFO 3009027 --- [ocp-async-2,75039de6eed03d6c,1d005e12b7d355c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:33:22.928 INFO 3009027 --- [ocp-schedules-15,4a42b92cf066ed73,360c22f61ddcc681] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:33:22.928+08:00, clusters=[1] 2025-04-08 20:33:22.929 INFO 3009027 --- [ocp-schedules-15,4a42b92cf066ed73,360c22f61ddcc681] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:33:22.941 INFO 3009027 --- [ocp-schedules-24,45f25e821911b3e3,68dfa48fb932cfa3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:33:22.941+08:00, clusters=[1] 2025-04-08 20:33:22.942 INFO 3009027 --- [ocp-schedules-24,45f25e821911b3e3,68dfa48fb932cfa3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:33:22.945 INFO 3009027 --- [ocp-schedules-25,efd139ec3e9e408b,687563533086a51b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:33:22.951 INFO 3009027 --- [ocp-schedules-14,efbfd87868826d29,25a9805fa2265e2c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:33:22.955 WARN 3009027 --- [ocp-async-4,9554afd969f0e32f,1e2bea00cf068c43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:22.955 INFO 3009027 --- [ocp-async-9,28115d489601f86d,7f5c329be7977794] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:22.964 INFO 3009027 --- [ocp-schedules-19,432045cb968a165c,2903fe0e150bffb0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:33:22.967 INFO 3009027 --- [ocp-schedules-16,5925f2c639707423,3f842944cc240d45] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:33:22.967 INFO 3009027 --- [ocp-schedules-16,f0bde7b5e1d19615,c16a84874a73249e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:33:22.968 INFO 3009027 --- [ocp-schedules-16,f0bde7b5e1d19615,c16a84874a73249e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:22.973 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:33:22.973 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:33:22.974 INFO 3009027 --- [ocp-schedules-13,464169b8f1c15aa2,5a9b5777f4fcc25d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:33:22.974 INFO 3009027 --- [ocp-schedules-13,464169b8f1c15aa2,5a9b5777f4fcc25d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:33:22.976 INFO 3009027 --- [ocp-schedules-26,fd608613e318fde1,d4b744370306a079] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:33:22.976 INFO 3009027 --- [ocp-schedules-26,fd608613e318fde1,d4b744370306a079] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:33:22.979 INFO 3009027 --- [ocp-schedules-13,464169b8f1c15aa2,5a9b5777f4fcc25d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:33:22.979 INFO 3009027 --- [ocp-schedules-13,464169b8f1c15aa2,5a9b5777f4fcc25d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:33:22.980 INFO 3009027 --- [ocp-schedules-1,ca2602eed4bb6d53,1c09bc76998c3980] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:33:22.983 INFO 3009027 --- [ocp-schedules-5,2ad9a44c353cb68b,54253b687f71adce] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:33:22.984 INFO 3009027 --- [ocp-schedules-5,2ad9a44c353cb68b,54253b687f71adce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:22.997 INFO 3009027 --- [ocp-schedules-28,647b6097d55cc938,dc08c4fc40084045] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:33:22.997 INFO 3009027 --- [ocp-schedules-28,647b6097d55cc938,dc08c4fc40084045] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:33:22.999 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:33:23.000 INFO 3009027 --- [ocp-schedules-4,8c5ce726ac052d02,9d69d7b3d823befe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:33:23.000 ERROR 3009027 --- [ocp-schedules-5,2ad9a44c353cb68b,54253b687f71adce] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:33:23.000 INFO 3009027 --- [ocp-schedules-5,2ad9a44c353cb68b,54253b687f71adce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:33:23.001 INFO 3009027 --- [ocp-schedules-4,8c5ce726ac052d02,9d69d7b3d823befe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:33:23.001 INFO 3009027 --- [ocp-schedules-4,8c5ce726ac052d02,9d69d7b3d823befe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:33:23.056 INFO 3009027 --- [ocp-schedules-20,f3532c49173e23fb,7f67a4fe68ce80bf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:33:23.087 INFO 3009027 --- [ocp-schedules-18,6872b88e58aa87a8,18555d7a63cd6e49] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:33:23.087 INFO 3009027 --- [ocp-schedules-18,6872b88e58aa87a8,18555d7a63cd6e49] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:33:23.088 INFO 3009027 --- [ocp-schedules-18,6872b88e58aa87a8,18555d7a63cd6e49] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:33:23.088 INFO 3009027 --- [ocp-schedules-18,6872b88e58aa87a8,18555d7a63cd6e49] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198297748959232 2025-04-08 20:33:23.088 INFO 3009027 --- [ocp-schedules-18,6872b88e58aa87a8,18555d7a63cd6e49] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:33:23.088 INFO 3009027 --- [ocp-schedules-18,6872b88e58aa87a8,18555d7a63cd6e49] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:33:23.124 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:23.124 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:33:23.176846+08:00, lastProcessTime=2025-04-08T20:33:13.176367+08:00 2025-04-08 20:33:23.335 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:33:23.176846+08:00 2025-04-08 20:33:23.775 INFO 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:33:23.775 INFO 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:33:23.778 INFO 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:33:23.780 WARN 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:33:23.780 INFO 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:33:23.781 WARN 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:33:23.785 INFO 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:33:23.786 WARN 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:33:23.787 ERROR 3009027 --- [ocp-schedules-19,52f6f7a78fc08563,4ee4f7a07619cc14] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:33:23.821 INFO 3009027 --- [ocp-schedules-30,7ccbeb76bf7cfa03,7365ad262e5c2207] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:33:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,e38d2909252bc38f,5fd579072e756746] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:24.494 INFO 3009027 --- [ocp-schedules-22,f59261c0e3da9a6c,8a5d0dda1714c5b9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:33:24.520 INFO 3009027 --- [ocp-schedules-22,f59261c0e3da9a6c,8a5d0dda1714c5b9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:33:24.520 INFO 3009027 --- [ocp-schedules-22,f59261c0e3da9a6c,8a5d0dda1714c5b9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:33:24.822 INFO 3009027 --- [ocp-schedules-29,cb80aa241df34dad,e17f26caf2fc1132] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:33:24.955 WARN 3009027 --- [ocp-async-9,5caddfd382728d8b,db9e380ea5319b9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:24.956 INFO 3009027 --- [ocp-async-3,cf64e6c739e8fa2b,45eeb08eedc36667] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:26.124 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:26.125 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:26.382 INFO 3009027 --- [ocp-schedules-29,969a6db184ff05c6,7fd04b7bc12e42e5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:33:26.383 INFO 3009027 --- [ocp-schedules-29,969a6db184ff05c6,7fd04b7bc12e42e5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:33:26.881 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:33:26.881 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:33:26.886 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:33:26.886 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:33:26.886 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:33:26.886 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:33:26.892 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:33:26.892 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:33:26.892 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:33:26.893 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:33:26.917 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:33:26.922 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:33:26.922 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:33:26.922 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:33:26.922 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:33:26.922 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:33:26.927 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:33:26.956 WARN 3009027 --- [ocp-async-5,be0015a8658a55bd,4c565381ebc7acfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:26.956 INFO 3009027 --- [ocp-async-6,8ac95afce0e94c46,ede436551506e913] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:33:27.225 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:33:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,dea258cd182ccc60,bd765df753ce9832] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:27.598 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.705886194S 2025-04-08 20:33:27.598 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:33:27.659 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:33:27.660 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:33:27.685 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:33:27.728 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:33:27.728 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:33:27.728 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:33:27.728 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:33:27.728 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:33:27.728 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:33:27.735 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:33:27.860 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] o.s.boot.SpringApplication : Started application in 0.973 seconds (JVM running for 649.477) 2025-04-08 20:33:27.860 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.973523993S 2025-04-08 20:33:28.138 INFO 3009027 --- [ocp-schedules-22,deec1fd5c14097cf,757bafc8a17d3714] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:33:28.956 WARN 3009027 --- [ocp-async-2,9bb2502d487345af,3c9f7534180cd3e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:28.956 INFO 3009027 --- [ocp-async-4,8365d7dbabb0b502,10f7dadb8214b178] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:29.125 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:29.126 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,28d6a0270d44e393,e02d856b86cc1eab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:30.956 WARN 3009027 --- [ocp-async-4,49cf67dcd9322f5c,328082a25ce8a67e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:30.956 INFO 3009027 --- [ocp-async-9,c068bb3ef9cee836,cdc53f30e2ccc2dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:32.126 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:32.127 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:32.897 INFO 3009027 --- [ocp-schedules-8,7b3e02cf4faa3acd,0470e66eeb78e68f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:33:32.898 INFO 3009027 --- [ocp-schedules-6,06162e911139ca57,de67e6af9257851e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:33:32.924 INFO 3009027 --- [ocp-async-5,5cf3a059cfda2091,7a5f3a934c443ab8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:33:32.946 INFO 3009027 --- [ocp-schedules-28,5ff780c880cc8815,c32735cd6cea9ddd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:33:32.956 WARN 3009027 --- [ocp-async-2,67ef169d2a3e1c59,a81b16571c586dbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:32.956 INFO 3009027 --- [ocp-async-1,c0c43922934f27af,a330bed3f1d6b32f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:33:33.176756+08:00, lastProcessTime=2025-04-08T20:33:23.176846+08:00 2025-04-08 20:33:33.312 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:33:33.176756+08:00 2025-04-08 20:33:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,4ae14eec97d54b7c,0c2a4403186e0ec2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:34.930 INFO 3009027 --- [ocp-schedules-18,1f2d26e607cc1bc7,cc15028706bfa441] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:33:34.956 INFO 3009027 --- [ocp-async-10,b0e718b0e87e9329,ed2ca1230e6617bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:34.956 WARN 3009027 --- [ocp-async-4,4fbd11446e3692e5,d9ca4e4afd603bd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:35.127 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:35.128 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:36.403 INFO 3009027 --- [ocp-schedules-14,d7867b229ea0a028,0e771ecb355b2fe2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:33:36.404 INFO 3009027 --- [ocp-schedules-14,d7867b229ea0a028,0e771ecb355b2fe2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:33:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,a4bc2569279dad1a,79d43f619083bda8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:36.956 WARN 3009027 --- [ocp-async-3,4249879b02322e41,19ef7373b2eba9a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:36.956 INFO 3009027 --- [ocp-async-10,b75ee79a606fc92f,207a9d263c42d55e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:37.067 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c23a4dfc1f2af8f2,055c876ae28b7344] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.11.12.102, traceId c23a4dfc1f2af8f2] 2025-04-08 20:33:37.079 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c23a4dfc1f2af8f2,055c876ae28b7344] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.11.12.102, traceId=c23a4dfc1f2af8f2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2025-04-08 20:33:37.135 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c23a4dfc1f2af8f2,055c876ae28b7344] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 26, current state: FAILED, new state: READY 2025-04-08 20:33:37.151 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c23a4dfc1f2af8f2,055c876ae28b7344] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.11.12.102, traceId=c23a4dfc1f2af8f2, duration=87 ms] 2025-04-08 20:33:37.151 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,c23a4dfc1f2af8f2,055c876ae28b7344] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=2, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=null, startTime=2025-04-08T20:33:37.078+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 20:33:37.326 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,6cfc750687ace029,faf60c00aad23ee5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=6cfc750687ace029, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:33:37.326 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,6cfc750687ace029,faf60c00aad23ee5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=6cfc750687ace029, duration=3 ms] 2025-04-08 20:33:37.327 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,d88f42a5c6760318,226bce569622021b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=d88f42a5c6760318, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:33:37.337 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,d88f42a5c6760318,226bce569622021b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=d88f42a5c6760318, duration=14 ms] 2025-04-08 20:33:37.749 INFO 3009027 --- [subtask-coordinator1,1308668168866431,dd5a61fc78f6c9ce] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:RETRY, state: RUNNING 2025-04-08 20:33:37.758 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:33:37.801 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,65f8a0bb17d12daf,62f56a40dbc16da4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=65f8a0bb17d12daf, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:33:37.800 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=26, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:33:37.744+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=26}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:33:37.809 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,65f8a0bb17d12daf,62f56a40dbc16da4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=65f8a0bb17d12daf, duration=12 ms] 2025-04-08 20:33:37.808 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 20:33:37.821 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:33:37.829 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:33:37.837 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:37.857 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:37.897 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:37.919 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:37.924 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:33:37.953 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:37.956 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:37.994 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:38.023 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.030 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:33:38.058 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.061 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.099 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:38.118 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.121 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:33:38.128 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:38.128 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:38.150 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.197 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.206 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 20:33:38.220 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:38.240 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.309 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:33:38.748 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.828 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:38.830 WARN 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:33:38.916 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:33:38.918 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 20:33:38.932 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,32f57d7de3797865,cbb7805ec8155dfe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=32f57d7de3797865, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:33:38.942 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,32f57d7de3797865,cbb7805ec8155dfe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=32f57d7de3797865, duration=13 ms] 2025-04-08 20:33:38.929 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:38.956 WARN 3009027 --- [ocp-async-6,9d39df48e80d1fbb,3cdde30eff85d6a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:38.957 INFO 3009027 --- [ocp-async-2,22fd3db186860e90,45269a90757dd018] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:38.962 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:39.008 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:33:39.361 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,e36a1a6223d575d1,5b6da7f568c3737e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=e36a1a6223d575d1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:33:39.364 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,e36a1a6223d575d1,5b6da7f568c3737e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=e36a1a6223d575d1, duration=7 ms] 2025-04-08 20:33:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,0fd72078843d3441,efddeee7b81b46d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:39.374 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:39.526 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:39.530 WARN 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 20:33:39.532 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 20:33:39.538 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 20:33:39.585 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 20:33:39.587 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:33:39.606 WARN 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:33:39.608 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:33:39.610 WARN 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:33:39.612 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:33:39.614 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:33:39.623 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:39.642 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:39.647 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:39.713 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:39.753 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:33:39.864 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:40.001 INFO 3009027 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:33:40.001 INFO 3009027 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:33:39.941 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:40.030 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:40.088 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:40.129 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:33:40.175 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:40.208 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:40.223 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 20:33:40.229 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:33:40.246 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:33:40.287 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 20:33:40.338 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:33:40.385 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:33:40.431 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 20:33:40.475 ERROR 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 20:33:40.529 ERROR 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:33:40.533 WARN 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:33:40.586 INFO 3009027 --- [manual-subtask-executor13,73ac1c39155401d0,ca1652567020cceb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:RETRY, state: FAILED 2025-04-08 20:33:40.938 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,9b5168befcb5de30,7c6f262f9793b8c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=9b5168befcb5de30, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:33:40.948 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,9b5168befcb5de30,7c6f262f9793b8c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=9b5168befcb5de30, duration=15 ms] 2025-04-08 20:33:40.957 WARN 3009027 --- [ocp-async-7,413dbb65a2881504,829067fa582f690e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:40.957 INFO 3009027 --- [ocp-async-4,c25102db8000eecf,4d189445991486a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:41.129 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:41.129 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:41.137 INFO 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, createTime=2025-04-08T20:29:48+08:00, startTime=2025-04-08T20:29:48.360+08:00, endTime=2025-04-08T20:30:08.855+08:00, subtasks=[SubtaskInstanceEntity{id=31, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=12, startTime=null, endTime=null}, SubtaskInstanceEntity{id=12, name=Attach cluster attributes, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.AttachClusterAttributesTask, seriesId=14, startTime=null, endTime=null}, SubtaskInstanceEntity{id=2, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=18, name=Update host arch info, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateHostSystemInfoTask, seriesId=7, startTime=2025-04-08T20:29:52.261+08:00, endTime=2025-04-08T20:29:52.748+08:00}, SubtaskInstanceEntity{id=3, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=16, startTime=2025-04-08T20:30:05.536+0 2025-04-08 20:33:41.147 WARN 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:33:41.147 INFO 3009027 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2025-04-08 20:33:41.173 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,32d25d461f7f53a6,116ae765add00c22] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=32d25d461f7f53a6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:33:41.176 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,32d25d461f7f53a6,116ae765add00c22] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=32d25d461f7f53a6, duration=7 ms] 2025-04-08 20:33:41.824 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,7b238bf87f6903b0,e050395d16b14e2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=7b238bf87f6903b0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:33:41.826 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,7b238bf87f6903b0,e050395d16b14e2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=7b238bf87f6903b0, duration=7 ms] 2025-04-08 20:33:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,efc555fdf99dff31,f5cfd2186aeda7dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:42.924 INFO 3009027 --- [ocp-async-4,14da369b3711449a,47eed3472cd8f2bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:33:42.947 INFO 3009027 --- [ocp-schedules-18,06f90c07aee50f2c,f6dad2e1a119eba8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:33:42.957 INFO 3009027 --- [ocp-async-3,385cadc7438d5c53,5ce33a4bac1c2e6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:42.957 WARN 3009027 --- [ocp-async-9,a9dd5e7303d94b24,267af92ee87f9b29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:33:43.176268+08:00, lastProcessTime=2025-04-08T20:33:33.176756+08:00 2025-04-08 20:33:43.196 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,1f4de798e0ae79d3,813cd305576097b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=1f4de798e0ae79d3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:33:43.205 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,1f4de798e0ae79d3,813cd305576097b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=1f4de798e0ae79d3, duration=13 ms] 2025-04-08 20:33:43.264 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:33:43.176268+08:00 2025-04-08 20:33:44.129 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:44.130 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:44.522 INFO 3009027 --- [ocp-schedules-3,6420c6394efc7ec2,32e341da92fc3005] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:33:44.572 INFO 3009027 --- [ocp-schedules-3,6420c6394efc7ec2,32e341da92fc3005] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:33:44.572 INFO 3009027 --- [ocp-schedules-3,6420c6394efc7ec2,32e341da92fc3005] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:33:44.951 INFO 3009027 --- [ocp-schedules-10,e631e757c7fbe54d,30e0349e85e2aff1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:33:44.957 WARN 3009027 --- [ocp-async-5,2bca1fc651b736fb,3c968cca75b9de17] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:44.957 INFO 3009027 --- [ocp-async-3,c79ad3eb7828aa36,69ac6f4ebfd20d8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,7d5d6a91e4f098ab,1067cf57d183bf68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:46.480 INFO 3009027 --- [ocp-schedules-24,b56f5300aa882b71,450bb5bf46c6e2e2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:33:46.481 INFO 3009027 --- [ocp-schedules-24,b56f5300aa882b71,450bb5bf46c6e2e2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:33:46.957 WARN 3009027 --- [ocp-async-1,095fcbd968462c63,93514d71090bbd70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:46.957 INFO 3009027 --- [ocp-async-7,6a5f91fc396850c7,aab388bd0545c5c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:47.130 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:47.131 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,027969741bbd8b74,48ec3f944ab7a612] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:48.957 WARN 3009027 --- [ocp-async-6,c03e0a5b67d546c2,9af50c76c6d2a2d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:48.957 INFO 3009027 --- [ocp-async-7,79b2634168bdc51f,b831c5928ac60668] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:50.131 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:50.132 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:50.957 WARN 3009027 --- [ocp-async-5,a206ca449d931164,82f33d3f32b5a121] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:50.957 INFO 3009027 --- [ocp-async-2,34185e9b37f7cf7d,5d50270763e99d48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,d80b0fe3551f8565,f9c759a2741b177d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:52.896 INFO 3009027 --- [ocp-schedules-22,da3686da3b631118,180e3cbd62b918b6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:33:52.896 INFO 3009027 --- [ocp-schedules-15,169669a746baad21,2239c05bb255ba3b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:33:52.896 INFO 3009027 --- [ocp-schedules-3,f348b0af07a48639,393ab20c82f76ee9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:33:52.897 INFO 3009027 --- [ocp-schedules-15,169669a746baad21,2239c05bb255ba3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:52.897 INFO 3009027 --- [ocp-schedules-3,f348b0af07a48639,393ab20c82f76ee9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:33:52.899 INFO 3009027 --- [ocp-schedules-23,99915f959ae7b1c1,0ef24e352096c539] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:33:52.899 INFO 3009027 --- [ocp-schedules-23,99915f959ae7b1c1,0ef24e352096c539] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:33:52.900 INFO 3009027 --- [ocp-schedules-4,119a6164ed2d4356,7a4b13ea696a539b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:33:52.900 INFO 3009027 --- [ocp-schedules-9,8c0c4732a60ad4b5,385dbacc7521c0dd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:33:52.900 INFO 3009027 --- [ocp-schedules-4,119a6164ed2d4356,7a4b13ea696a539b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:33:52.900 INFO 3009027 --- [ocp-schedules-9,8c0c4732a60ad4b5,385dbacc7521c0dd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:33:52.906 INFO 3009027 --- [ocp-schedules-21,9948b4af2d2facf3,9b9afd9766c34b59] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:33:52.924 INFO 3009027 --- [ocp-async-2,6879e5ed114f8211,5cc03d0ece6bc2c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:33:52.934 INFO 3009027 --- [ocp-schedules-20,b5f8075d43700140,e2618896708fb8b1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:33:52.935 INFO 3009027 --- [ocp-schedules-20,b5f8075d43700140,e2618896708fb8b1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:52.935 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:33:52.949 INFO 3009027 --- [ocp-schedules-6,bfdaac667fa809ff,6b80eb68828f1cfd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:33:52.958 WARN 3009027 --- [ocp-async-8,4137cb5b40618327,ecd1481acb772fe3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:52.958 INFO 3009027 --- [ocp-async-8,42451e2c57753c79,0473035e920e0b4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:52.966 INFO 3009027 --- [ocp-schedules-2,78cf59dff70e76d6,70135766c5359fc9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:33:52.973 INFO 3009027 --- [ocp-schedules-16,f0bde7b5e1d19615,c16a84874a73249e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:33:52.973 INFO 3009027 --- [ocp-schedules-16,bdddb5b9e7d2b438,58acef02ce585c51] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:33:52.974 INFO 3009027 --- [ocp-schedules-16,bdddb5b9e7d2b438,58acef02ce585c51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:33:52.979 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:33:52.979 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:33:53.132 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:53.132 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:33:53.176346+08:00, lastProcessTime=2025-04-08T20:33:43.176268+08:00 2025-04-08 20:33:53.264 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:33:53.176346+08:00 2025-04-08 20:33:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,712b60a06c8247a2,af3a9b75982a8b1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:54.958 WARN 3009027 --- [ocp-async-6,6faec5cb963e0169,fdee6cb287417b4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:54.958 INFO 3009027 --- [ocp-async-9,0a357c6f4dcd1e40,ee0c5d2ed387d541] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:54.983 INFO 3009027 --- [ocp-schedules-4,ea9f17f3d3baf2fe,68260a8ab3347554] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:33:56.132 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:56.133 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:33:56.555 INFO 3009027 --- [ocp-schedules-22,0647bcbce621bfcf,e716203fe9b54ec3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:33:56.556 INFO 3009027 --- [ocp-schedules-22,0647bcbce621bfcf,e716203fe9b54ec3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:33:56.958 WARN 3009027 --- [ocp-async-2,54479c6cad83a256,b4597e7a10e59290] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:56.958 INFO 3009027 --- [ocp-async-7,c8f26fb5f5546554,eea966057d3c7c97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,e60117d14d34a279,bfc8cbfccf6f02ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115640, interval=60 2025-04-08 20:33:58.958 WARN 3009027 --- [ocp-async-7,46ac29aa3ebfdebd,f475497f94fb0522] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:33:58.958 INFO 3009027 --- [ocp-async-3,31f012debdbd9205,c99b216cd23c9eb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:33:59.133 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:33:59.134 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,4f4d61a8c7cc8158,c69d4b8a57963fb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:00.958 WARN 3009027 --- [ocp-async-8,4c29b65caeb7849c,18222c2aea7bcdb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:00.958 INFO 3009027 --- [ocp-async-3,a97a829eea3011ac,9a99eafdfd840f50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:02.134 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:02.135 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:02.897 INFO 3009027 --- [ocp-schedules-15,169669a746baad21,2239c05bb255ba3b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:34:02.898 INFO 3009027 --- [ocp-schedules-3,f348b0af07a48639,393ab20c82f76ee9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:34:02.924 INFO 3009027 --- [ocp-async-3,d495a933131b82bc,350a30bace3c8da1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:34:02.936 INFO 3009027 --- [ocp-schedules-20,b5f8075d43700140,e2618896708fb8b1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:34:02.950 INFO 3009027 --- [ocp-schedules-21,3e717fcd99703369,fa3666c9b00ba6c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:34:02.958 WARN 3009027 --- [ocp-async-5,f5495879ac1b8615,285cc0a97761b9a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:02.958 INFO 3009027 --- [ocp-async-2,abb0dd1905576f27,453b229f27397df0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:34:03.177551+08:00, lastProcessTime=2025-04-08T20:33:53.176346+08:00 2025-04-08 20:34:03.266 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:34:03.177551+08:00 2025-04-08 20:34:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,1381f7d8e3d3e396,d93e8b4c50ba9269] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:04.574 INFO 3009027 --- [ocp-schedules-21,dcac8994c09185e8,eb7a9569235313a6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:34:04.649 INFO 3009027 --- [ocp-schedules-21,dcac8994c09185e8,eb7a9569235313a6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:34:04.649 INFO 3009027 --- [ocp-schedules-21,dcac8994c09185e8,eb7a9569235313a6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:34:04.958 WARN 3009027 --- [ocp-async-2,ab5bc3e28f4379fe,78597403fbd9fb9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:04.959 INFO 3009027 --- [ocp-async-4,f047940499661432,0fee09393f01fde0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:05.005 INFO 3009027 --- [ocp-schedules-29,801e57b9361f80c8,544d1c76def585f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:34:05.135 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:05.136 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,93cdc184b47f58f9,070dd0d3757fa6ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:06.560 INFO 3009027 --- [ocp-schedules-9,034ba69805a186cb,5c55fd8144506147] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:34:06.561 INFO 3009027 --- [ocp-schedules-9,034ba69805a186cb,5c55fd8144506147] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:34:06.959 WARN 3009027 --- [ocp-async-6,43ce3c3b9a0824cd,c92051089dd2a22d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:06.959 INFO 3009027 --- [ocp-async-4,f5920ccee80ee4cb,796a3586962eaf38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:08.136 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:08.137 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:08.959 WARN 3009027 --- [ocp-async-5,8cf171fc22d1d315,077421d27c55e93d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:08.959 INFO 3009027 --- [ocp-async-4,3d17e7bc799c397d,dafd95accf32c889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,b701e0e15fa0699e,c4da725debb551f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:10.002 INFO 3009027 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:34:10.002 INFO 3009027 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:34:10.959 WARN 3009027 --- [ocp-async-2,913e37e9cd5c9b57,245770321de95650] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:10.959 INFO 3009027 --- [ocp-async-4,782f8abe2ac8d923,a78eef1bdfc7278d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:10.991 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:34:11.137 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:11.137 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,2bf0b57b945e989f,7652267d13375385] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:12.408 INFO 3009027 --- [prometheus-cache-monitor1,866e55e134a16ddd,65ce01686c4c5add] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1664,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":414} 2025-04-08 20:34:12.408 INFO 3009027 --- [prometheus-cache-monitor1,866e55e134a16ddd,65ce01686c4c5add] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:34:12.924 INFO 3009027 --- [ocp-async-4,c31c4bd158f6c029,0c33e0cb56d2f221] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:34:12.952 INFO 3009027 --- [ocp-schedules-22,13151507ebf663bd,de94ad7dbdaf13b7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:34:12.959 WARN 3009027 --- [ocp-async-10,20b3b24eb615bbf9,0e3527edf1696f69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:12.959 INFO 3009027 --- [ocp-async-6,3a2b18d841f2f230,d29fa9394a89c83b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:34:13.177248+08:00, lastProcessTime=2025-04-08T20:34:03.177551+08:00 2025-04-08 20:34:13.297 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:34:13.177248+08:00 2025-04-08 20:34:14.138 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:14.138 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:14.959 WARN 3009027 --- [ocp-async-6,0a7cacbab22dc213,29dce5284744d891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:14.959 INFO 3009027 --- [ocp-async-7,4f5c743202a435b9,b55df104d052537c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:15.041 INFO 3009027 --- [ocp-schedules-12,49543bf26f0aad3f,fea33aa1aa3d081c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:34:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,df98dd95702171f3,f1c03ef5e1fa689e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:15.655 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115580, nodeCount=84, affectRows=84 2025-04-08 20:34:15.656 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115580, nodeCount=82, validCount=82 2025-04-08 20:34:15.703 INFO 3009027 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115580, nodeCount=78, affectRows=78 2025-04-08 20:34:15.703 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115580, nodeCount=91, affectRows=91 2025-04-08 20:34:15.703 INFO 3009027 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115580, nodeCount=74, affectRows=74 2025-04-08 20:34:15.704 INFO 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115580, nodeCount=91, affectRows=91 2025-04-08 20:34:15.704 INFO 3009027 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115580, nodeCount=77, validCount=77 2025-04-08 20:34:15.704 INFO 3009027 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115580, nodeCount=74, validCount=74 2025-04-08 20:34:15.704 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115580, nodeCount=91, validCount=91 2025-04-08 20:34:15.704 INFO 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115580, nodeCount=91, validCount=91 2025-04-08 20:34:16.571 INFO 3009027 --- [ocp-schedules-25,0b1643a6803f4afb,7db99115b01b128b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:34:16.572 INFO 3009027 --- [ocp-schedules-25,0b1643a6803f4afb,7db99115b01b128b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:34:16.959 WARN 3009027 --- [ocp-async-7,f1d5d36dff31d7a7,801b57c3997b8a0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:16.959 INFO 3009027 --- [ocp-async-8,1411ca070942f6e9,08017ebd17d06a4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:17.139 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:17.139 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,51cb1bbfd152071d,7e52aa6904b59f3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:18.960 WARN 3009027 --- [ocp-async-10,492a502a13ad79a8,aa0a18fd65878991] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:18.960 INFO 3009027 --- [ocp-async-3,1ea234906052b9e8,f93fef161c98f067] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:20.139 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:20.140 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:20.960 WARN 3009027 --- [ocp-async-6,3e57a20413feb79b,ee7150f3f508777c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:20.960 INFO 3009027 --- [ocp-async-3,54f7af474c6c4ac9,28c71cb0263d026a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,d29ba5a22d98a792,3e00cc0b83855103] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:22.895 INFO 3009027 --- [ocp-schedules-5,76abf10ceca9ea3c,c16aebe59aa9e7c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:34:22.896 INFO 3009027 --- [ocp-schedules-20,7d57b62882f6f651,82e68962cb131bda] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:34:22.896 INFO 3009027 --- [ocp-schedules-30,aef4ea2cc23e17b7,a6b3d4f329688325] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:34:22.896 INFO 3009027 --- [ocp-schedules-28,7f9d2938f39092ac,9085220f3d45233a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:34:22.896 INFO 3009027 --- [ocp-schedules-21,512842bb7431e786,e3322532bdb1c3b2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:34:22.897 INFO 3009027 --- [ocp-schedules-30,aef4ea2cc23e17b7,a6b3d4f329688325] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.897 INFO 3009027 --- [ocp-schedules-21,512842bb7431e786,e3322532bdb1c3b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:34:22.898 INFO 3009027 --- [ocp-schedules-23,928007e2e1b6faac,3455223119f0030c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:34:22.898 INFO 3009027 --- [ocp-schedules-3,4ca021088f11a264,e655d20728ccb023] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:34:22.898 INFO 3009027 --- [ocp-schedules-23,928007e2e1b6faac,3455223119f0030c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:34:22.898 INFO 3009027 --- [ocp-schedules-3,4ca021088f11a264,e655d20728ccb023] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:34:22.898 INFO 3009027 --- [ocp-schedules-19,fe30eb5fa40d4e7d,22a98ac6bdbfa98c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:34:22.898 INFO 3009027 --- [ocp-schedules-19,fe30eb5fa40d4e7d,22a98ac6bdbfa98c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:34:22.902 INFO 3009027 --- [ocp-schedules-4,7b59a35f57df502d,c3102cfcf2942532] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 20:34:22.902 INFO 3009027 --- [ocp-schedules-4,7b59a35f57df502d,c3102cfcf2942532] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 20:34:22.903 INFO 3009027 --- [ocp-schedules-28,7f9d2938f39092ac,9085220f3d45233a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:34:22.913 INFO 3009027 --- [ocp-schedules-8,9545b0ee3becb0cc,96ba12fd63e09821] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1666,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:34:22.913 INFO 3009027 --- [ocp-schedules-8,9545b0ee3becb0cc,96ba12fd63e09821] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:34:22.914 INFO 3009027 --- [pool-overview-container-7,e2920a6950361557,2bd7e778a08098d7] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:34:22.914 INFO 3009027 --- [pool-overview-container-6,e33af1163a181505,d160417bd25fa621] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:34:22.917 INFO 3009027 --- [ocp-schedules-1,f118a3c2fa5851f1,4572303d6f4030ef] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:34:22.917 INFO 3009027 --- [ocp-schedules-1,f118a3c2fa5851f1,4572303d6f4030ef] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:34:22.918 INFO 3009027 --- [ocp-schedules-25,7acdb5de06d0eafd,80efd255c0803b7a] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:34:22.919 INFO 3009027 --- [pool-11-thread-11,e33af1163a181505,d9b5bfbd322dc9a6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 20:34:22.919 INFO 3009027 --- [pool-11-thread-9,e33af1163a181505,862faa0b24ebce82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 20:34:22.919 INFO 3009027 --- [pool-11-thread-12,e33af1163a181505,15c8466e974ebbc7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 20:34:22.920 INFO 3009027 --- [pool-11-thread-10,e33af1163a181505,4f11cb6f5fe1b366] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 20:34:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:34:22.924 INFO 3009027 --- [ForkJoinPool-7-worker-9,e33af1163a181505,40b413293c9f80a5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 20:34:22.924 INFO 3009027 --- [ForkJoinPool-7-worker-9,e33af1163a181505,40b413293c9f80a5] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 20:34:22.924 INFO 3009027 --- [ForkJoinPool-7-worker-9,e33af1163a181505,40b413293c9f80a5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 20:34:22.925 INFO 3009027 --- [ocp-async-1,5942f1ac2c810681,b5e41241d8e472cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:34:22.925 INFO 3009027 --- [ocp-schedules-25,7acdb5de06d0eafd,80efd255c0803b7a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:34:22.926 INFO 3009027 --- [pool-11-thread-11,e33af1163a181505,d9b5bfbd322dc9a6] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 20:34:22.927 WARN 3009027 --- [ocp-schedules-25,7acdb5de06d0eafd,80efd255c0803b7a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:34:22.927 INFO 3009027 --- [ocp-schedules-25,7acdb5de06d0eafd,80efd255c0803b7a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:34:22.928 INFO 3009027 --- [ocp-schedules-5,76abf10ceca9ea3c,c16aebe59aa9e7c8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:34:22.928 WARN 3009027 --- [ocp-schedules-25,7acdb5de06d0eafd,80efd255c0803b7a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:34:22.932 INFO 3009027 --- [ocp-schedules-6,0c1c810a82131bb2,cdf852fa32ff5216] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:34:22.932 INFO 3009027 --- [ocp-schedules-2,c485af0993c1aa74,a3ecc11b1f716743] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:34:22.932 INFO 3009027 --- [ocp-schedules-25,7acdb5de06d0eafd,80efd255c0803b7a] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=1 2025-04-08 20:34:22.933 INFO 3009027 --- [ocp-schedules-6,0c1c810a82131bb2,cdf852fa32ff5216] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.934 INFO 3009027 --- [ocp-schedules-14,f62d18b20bc7f5da,207bd3023d24f21d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:34:22.934+08:00, clusters=[1] 2025-04-08 20:34:22.934 INFO 3009027 --- [ocp-schedules-14,f62d18b20bc7f5da,207bd3023d24f21d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:34:22.934 INFO 3009027 --- [pool-11-thread-10,e33af1163a181505,4f11cb6f5fe1b366] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 20:34:22.936 INFO 3009027 --- [pool-11-thread-10,e33af1163a181505,4f11cb6f5fe1b366] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 20:34:22.937 INFO 3009027 --- [pool-11-thread-9,e33af1163a181505,862faa0b24ebce82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 20:34:22.939 INFO 3009027 --- [ForkJoinPool-7-worker-9,e33af1163a181505,40b413293c9f80a5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:34:22.940 INFO 3009027 --- [ForkJoinPool-7-worker-9,e33af1163a181505,08c758ae8d0b2306] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 20:34:22.940 INFO 3009027 --- [ocp-schedules-23,dd6de17119c52a2e,1018f6f0084cc77e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 20:34:22.940 INFO 3009027 --- [pool-11-thread-12,e33af1163a181505,15c8466e974ebbc7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 20:34:22.941 INFO 3009027 --- [ocp-schedules-23,dd6de17119c52a2e,1018f6f0084cc77e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.946 INFO 3009027 --- [ocp-schedules-3,f9b0b9f0cb1dc236,79b3bdfeac418570] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:34:22.946+08:00, clusters=[1] 2025-04-08 20:34:22.947 INFO 3009027 --- [ocp-schedules-18,8324c05008fd309d,38ad7577bd70c7ab] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 20:34:22.947 INFO 3009027 --- [ocp-schedules-18,8324c05008fd309d,38ad7577bd70c7ab] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 20:34:22.947 INFO 3009027 --- [ocp-schedules-18,8324c05008fd309d,38ad7577bd70c7ab] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=419 2025-04-08 20:34:22.948 INFO 3009027 --- [ocp-schedules-3,f9b0b9f0cb1dc236,79b3bdfeac418570] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:34:22.953 INFO 3009027 --- [ocp-schedules-27,202b4b506f9add79,029a04f0581926d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:34:22.954 INFO 3009027 --- [ocp-schedules-19,4045736cb1e33c9d,387d0a66521c1598] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:34:22.955 INFO 3009027 --- [ocp-schedules-4,3cceb551c65a74dd,49c53070189975aa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:34:22.955 INFO 3009027 --- [ocp-schedules-4,3cceb551c65a74dd,49c53070189975aa] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:34:22.958 INFO 3009027 --- [ocp-schedules-28,3d1f5cfe74f1529e,72ee6761e3d458c3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:34:22.958 ERROR 3009027 --- [ocp-schedules-23,dd6de17119c52a2e,1018f6f0084cc77e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:34:22.958 INFO 3009027 --- [ocp-schedules-23,dd6de17119c52a2e,1018f6f0084cc77e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 20:34:22.958 INFO 3009027 --- [ocp-schedules-28,3d1f5cfe74f1529e,72ee6761e3d458c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.960 INFO 3009027 --- [ocp-async-8,103f0241af422cf9,5e07ad052609fd84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:22.960 WARN 3009027 --- [ocp-async-4,2b4c47e19456f501,a8cbf025c8163c95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:22.961 INFO 3009027 --- [pool-11-thread-9,e33af1163a181505,862faa0b24ebce82] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 20:34:22.966 INFO 3009027 --- [pool-22-thread-1,3cceb551c65a74dd,1294ddc2a6b548f3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 20:34:22.967 INFO 3009027 --- [pool-11-thread-9,e33af1163a181505,862faa0b24ebce82] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 20:34:22.967 INFO 3009027 --- [ocp-schedules-8,f1538a3486beda15,beaee807a0bc1899] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:34:22.972 ERROR 3009027 --- [ocp-schedules-28,3d1f5cfe74f1529e,72ee6761e3d458c3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:34:22.972 INFO 3009027 --- [ocp-schedules-28,3d1f5cfe74f1529e,72ee6761e3d458c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:34:22.973 WARN 3009027 --- [ocp-schedules-4,3cceb551c65a74dd,49c53070189975aa] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$ece00efc.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:34:22.973 INFO 3009027 --- [ocp-schedules-4,3cceb551c65a74dd,49c53070189975aa] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:34:22.978 INFO 3009027 --- [ocp-schedules-17,a15df20657256645,03300df434a1da7b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:34:22.978 INFO 3009027 --- [ocp-schedules-17,a15df20657256645,03300df434a1da7b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:34:22.979 INFO 3009027 --- [ocp-schedules-1,2e79322a55b4f1fa,02f98ee116cfde29] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:34:22.979 INFO 3009027 --- [ocp-schedules-1,2e79322a55b4f1fa,02f98ee116cfde29] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:34:22.980 INFO 3009027 --- [ocp-schedules-16,bdddb5b9e7d2b438,58acef02ce585c51] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:34:22.980 INFO 3009027 --- [ocp-schedules-16,602e26ac7d99939c,4fd91ed29b61b14e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:34:22.980 INFO 3009027 --- [ocp-schedules-16,602e26ac7d99939c,4fd91ed29b61b14e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.981 INFO 3009027 --- [ocp-schedules-10,be498f16385ae10b,f09ce8f5a5705c98] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:34:22.981 INFO 3009027 --- [ocp-schedules-18,8324c05008fd309d,38ad7577bd70c7ab] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=419 2025-04-08 20:34:22.981 INFO 3009027 --- [ocp-schedules-10,be498f16385ae10b,f09ce8f5a5705c98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:22.984 INFO 3009027 --- [ocp-schedules-1,2e79322a55b4f1fa,02f98ee116cfde29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:34:22.984 INFO 3009027 --- [ocp-schedules-1,2e79322a55b4f1fa,02f98ee116cfde29] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:34:22.985 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:34:22.985 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:34:22.992 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:34:22.993 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:34:22.998 INFO 3009027 --- [ocp-schedules-22,6c8acb7356c93769,fa163f3cb53887a5] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:34:22.998 INFO 3009027 --- [ocp-schedules-22,6c8acb7356c93769,fa163f3cb53887a5] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:34:23.000 INFO 3009027 --- [ocp-schedules-11,497127cbcd09cfd0,084d6b7c900a73cf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:34:23.000 INFO 3009027 --- [ocp-schedules-11,497127cbcd09cfd0,084d6b7c900a73cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:23.001 INFO 3009027 --- [ocp-schedules-13,20e18b3cf9186c9b,097f443b6784ba0d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:34:23.003 INFO 3009027 --- [ocp-schedules-13,20e18b3cf9186c9b,097f443b6784ba0d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:34:23.003 INFO 3009027 --- [ocp-schedules-13,20e18b3cf9186c9b,097f443b6784ba0d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:34:23.016 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:34:23.017 ERROR 3009027 --- [ocp-schedules-11,497127cbcd09cfd0,084d6b7c900a73cf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:34:23.017 INFO 3009027 --- [ocp-schedules-11,497127cbcd09cfd0,084d6b7c900a73cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:34:23.063 INFO 3009027 --- [ocp-schedules-2,b4c6a042cbc4b0c7,ff9f3a21ad32c0e4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:34:23.088 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:34:23.088 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:34:23.092 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:34:23.092 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198297748959232 2025-04-08 20:34:23.092 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=766199115910868992 2025-04-08 20:34:23.093 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:34:23.093 INFO 3009027 --- [ocp-schedules-14,1ba837f08e7a921d,0bc6c3816d97a9b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:34:23.140 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:23.141 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:34:23.176696+08:00, lastProcessTime=2025-04-08T20:34:13.177248+08:00 2025-04-08 20:34:23.330 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:34:23.176696+08:00 2025-04-08 20:34:23.790 INFO 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:34:23.790 INFO 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:34:23.793 INFO 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:34:23.794 WARN 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:34:23.795 INFO 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:34:23.796 WARN 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:34:23.799 INFO 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:34:23.801 WARN 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:34:23.801 ERROR 3009027 --- [ocp-schedules-9,2d3b09dea0147736,445577ad4132a4b5] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:34:23.903 INFO 3009027 --- [ocp-schedules-18,d219fc5f81a32131,c99ec6359e66b3c0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:34:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,064c9058304a72a3,b103bcd922bbd1be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:24.651 INFO 3009027 --- [ocp-schedules-8,29697067fc586ec1,eedf415d9ae18541] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:34:24.657 INFO 3009027 --- [ocp-schedules-8,29697067fc586ec1,eedf415d9ae18541] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:34:24.657 INFO 3009027 --- [ocp-schedules-8,29697067fc586ec1,eedf415d9ae18541] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:34:24.960 WARN 3009027 --- [ocp-async-5,520fd8e51c68e199,99a3dd53c9f9e4d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:24.960 INFO 3009027 --- [ocp-async-2,eec9eb7cf1a8ffe6,3fc5a67bb71544bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:25.066 INFO 3009027 --- [ocp-schedules-7,d2090889d5298318,7aaf85c52e4a0140] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:34:26.141 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:26.142 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:26.643 INFO 3009027 --- [ocp-schedules-3,386d3fb5b7e86a35,04fddc7c1c4c38ca] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:34:26.644 INFO 3009027 --- [ocp-schedules-3,386d3fb5b7e86a35,04fddc7c1c4c38ca] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:34:26.960 WARN 3009027 --- [ocp-async-9,c326511655596301,affb47547c6ef216] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:26.960 INFO 3009027 --- [ocp-async-3,8a5f648b9356eee4,aff9da391c2307c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:34:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,e9e830038a8bf8c8,0e3d07d8d9f1eafd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:28.960 WARN 3009027 --- [ocp-async-1,f4158005917d7bc0,d1da819f00d5b236] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:28.960 INFO 3009027 --- [ocp-async-10,8633f2eae2c4c813,b086506fa790a07d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:29.142 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:29.142 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,d53849db4ad5fe1e,be8a74edfcbb56cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:30.960 WARN 3009027 --- [ocp-async-5,f9c56aca8b9aa5af,b570ccdcfd4db5ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:30.961 INFO 3009027 --- [ocp-async-8,224d7ee8bdf85c3c,a17a07471615218f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:32.143 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:32.143 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:32.897 INFO 3009027 --- [ocp-schedules-30,aef4ea2cc23e17b7,a6b3d4f329688325] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:34:32.898 INFO 3009027 --- [ocp-schedules-21,512842bb7431e786,e3322532bdb1c3b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:34:32.925 INFO 3009027 --- [ocp-async-9,8c7f46f72a605cd1,5ca941ba78befd25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:34:32.933 INFO 3009027 --- [ocp-schedules-6,0c1c810a82131bb2,cdf852fa32ff5216] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 20:34:32.955 INFO 3009027 --- [ocp-schedules-23,2be65b163810e239,c7a97762fef2dd17] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:34:32.961 WARN 3009027 --- [ocp-async-7,d5dd4aaf5ac9e4b5,2b3815bb72f757be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:32.961 INFO 3009027 --- [ocp-async-1,f0b62f0ed132eb1c,21db732f092966c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:34:33.177759+08:00, lastProcessTime=2025-04-08T20:34:23.176696+08:00 2025-04-08 20:34:33.263 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:34:33.177759+08:00 2025-04-08 20:34:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,d17718ca220c4205,7dfba8a3f0570bab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:34.961 WARN 3009027 --- [ocp-async-1,7eb3dd7034ac5d28,8fceb9d8d05f4c7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:34.961 INFO 3009027 --- [ocp-async-10,0d77a00e8b67da5c,354d24f5b876a469] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:35.094 INFO 3009027 --- [ocp-schedules-9,256fbf732dc45dca,688d16385fcdeae1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:34:35.143 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:35.144 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,4011359396c609e4,282c49a2f60c2e03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:36.665 INFO 3009027 --- [ocp-schedules-9,ff09bc5476c7a10b,7f0bcb9f6f88178f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:34:36.666 INFO 3009027 --- [ocp-schedules-9,ff09bc5476c7a10b,7f0bcb9f6f88178f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:34:36.961 WARN 3009027 --- [ocp-async-8,cfda06e2d09a371b,3a5998ef6b44d51e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:36.961 INFO 3009027 --- [ocp-async-6,9904e797f528d3e4,6384df5cab85cbbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:38.144 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:38.145 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:38.961 WARN 3009027 --- [ocp-async-7,f91f0a8cc8ab9d08,efced0465ce0bd32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:38.961 INFO 3009027 --- [ocp-async-6,d9954403591dbedf,73e77d7631834aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,6d7582bd9095c1be,a8d20328ca4bf384] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:40.001 INFO 3009027 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:34:40.002 INFO 3009027 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:34:40.953 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:34:40.954 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:34:40.954 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:34:40.955 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:34:40.955 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:34:40.959 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1), status=TAKINGOVER 2025-04-08 20:34:40.959 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 20:34:40.961 WARN 3009027 --- [ocp-async-1,d5186b44a8a51484,82f8032802e5c5a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:40.961 INFO 3009027 --- [ocp-async-5,a3ef097c8b0036ec,8ac93d8cacae5598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:41.145 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:41.146 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,6648d9d8237c1b5c,866faae9f8610ce4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:42.925 INFO 3009027 --- [ocp-async-5,5f7af99d405f54d6,61ee666828199d11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:34:42.956 INFO 3009027 --- [ocp-schedules-22,f2b832a211d7478f,de491bc8382e2295] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:34:42.961 WARN 3009027 --- [ocp-async-9,0501ebce560f5883,4e615fea3d66167c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:42.961 INFO 3009027 --- [ocp-async-2,c2f143caf5d66583,4f9a9f08d844b339] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:34:43.177166+08:00, lastProcessTime=2025-04-08T20:34:33.177759+08:00 2025-04-08 20:34:43.265 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:34:43.177166+08:00 2025-04-08 20:34:44.143 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:34:13+08:00, expireTime=null, isLeader=false} 2025-04-08 20:34:44.144 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:34:44.146 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:44.146 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:44.658 INFO 3009027 --- [ocp-schedules-12,6812d2c888e1b466,0880b6bd26580f27] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:34:44.669 INFO 3009027 --- [ocp-schedules-12,6812d2c888e1b466,0880b6bd26580f27] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:34:44.669 INFO 3009027 --- [ocp-schedules-12,6812d2c888e1b466,0880b6bd26580f27] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:34:44.961 WARN 3009027 --- [ocp-async-3,302176803b126b91,427604ef98cd0eb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:44.962 INFO 3009027 --- [ocp-async-2,007a37ea46f75c83,c1eb9fa4fbd063b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:45.113 INFO 3009027 --- [ocp-schedules-23,e00b6e72d0804aa5,af8e0482fb5de919] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:34:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,fa41c31d13345cf1,0ac6fc729bb6057a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:46.788 INFO 3009027 --- [ocp-schedules-21,7febd99750697e29,a12d0adebe8b8724] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:34:46.790 INFO 3009027 --- [ocp-schedules-21,7febd99750697e29,a12d0adebe8b8724] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:34:46.962 WARN 3009027 --- [ocp-async-4,5cf56a3a7ccbc5e5,49d26877b7ea7432] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:46.962 INFO 3009027 --- [ocp-async-10,f555b2c10554cc82,071ed32d0481ab33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:47.146 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:47.147 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,213e52936d3ef0b7,785ca626dc6ea8e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:48.962 WARN 3009027 --- [ocp-async-5,ca393e2286e1e6f9,9b90c75788362a6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:48.962 INFO 3009027 --- [ocp-async-8,00292341c501e392,49503e1540e66bde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:50.147 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:50.148 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:50.962 WARN 3009027 --- [ocp-async-7,94a62813027ebe3e,e430eb5bcd53120e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:50.962 INFO 3009027 --- [ocp-async-3,49fd9d8d4527b1c8,5258b0e1264a0871] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,6385c6a1a7d472bb,4ee3bb553ce735bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:52.896 INFO 3009027 --- [ocp-schedules-3,23d69443a08b758c,105781531ff5761e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:34:52.896 INFO 3009027 --- [ocp-schedules-26,ecb4f5afe0a461f0,d06e3950254168c2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:34:52.897 INFO 3009027 --- [ocp-schedules-26,ecb4f5afe0a461f0,d06e3950254168c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:52.897 INFO 3009027 --- [ocp-schedules-6,efa140f579033b5d,704c017e28e76dfa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-6,efa140f579033b5d,704c017e28e76dfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-1,3820e547f90a9653,7185eba5969fa432] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-1,3820e547f90a9653,7185eba5969fa432] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:34:52.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-17,2343248e18922863,40382ac32fd9d403] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-17,2343248e18922863,40382ac32fd9d403] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-29,eb999dca39fc48ca,2e2694e6f6e61807] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:34:52.898 INFO 3009027 --- [ocp-schedules-29,eb999dca39fc48ca,2e2694e6f6e61807] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:34:52.925 INFO 3009027 --- [ocp-async-3,c7a23c69a52b3f0d,169ac4900872c388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:34:52.958 INFO 3009027 --- [ocp-schedules-24,95c389d8dee53113,bb357557c9e5765c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:34:52.962 WARN 3009027 --- [ocp-async-1,270b7143e3a19b4a,4bc8c0d68bff0b03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:52.962 INFO 3009027 --- [ocp-async-4,00a9a63db418d6d7,b2e623caa08c9e00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:52.968 INFO 3009027 --- [ocp-schedules-14,0b50ff54b09566b5,9b8095587bac367d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:34:52.982 INFO 3009027 --- [ocp-schedules-22,c8571acec1ead1eb,c24627026c0c222b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:34:52.985 INFO 3009027 --- [ocp-schedules-16,602e26ac7d99939c,4fd91ed29b61b14e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:34:52.986 INFO 3009027 --- [ocp-schedules-16,d1b136e5f440b676,538f57180564541c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:34:52.986 INFO 3009027 --- [ocp-schedules-16,d1b136e5f440b676,538f57180564541c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:34:52.991 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:34:52.991 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:34:53.148 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:53.149 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:34:53.177708+08:00, lastProcessTime=2025-04-08T20:34:43.177166+08:00 2025-04-08 20:34:53.266 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:34:53.177708+08:00 2025-04-08 20:34:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,616de1813dad8edf,a334e5d21623d957] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:54.962 WARN 3009027 --- [ocp-async-9,f45bcf733287ec0f,3a84640cc540acaa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:54.962 INFO 3009027 --- [ocp-async-5,3f27fe2fbe23e21e,5fe179d18e93bc67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:55.140 INFO 3009027 --- [ocp-schedules-5,4674ac6bdf64d947,69ab8c5f052d9b95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:34:56.149 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:56.150 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:34:56.798 INFO 3009027 --- [ocp-schedules-12,44cb549c1a2cd5bd,be50256047e3c6c8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:34:56.799 INFO 3009027 --- [ocp-schedules-12,44cb549c1a2cd5bd,be50256047e3c6c8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:34:56.962 WARN 3009027 --- [ocp-async-6,d29605eeff11cffc,2ed29bdba6e96305] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:56.962 INFO 3009027 --- [ocp-async-7,195674b1b3fe170e,fcd4181c5ca4e2d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,afa0beac3656f5bf,c06cfc3456c7b005] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115700, interval=60 2025-04-08 20:34:58.962 WARN 3009027 --- [ocp-async-10,f8cb338d32179fb3,2240b65742c6814a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:34:58.962 INFO 3009027 --- [ocp-async-1,71168348f250224c,73a34bb1a0b55645] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:34:59.150 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:34:59.150 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:00.403 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:30:01.0, nextRun=2025-04-08 20:35:00.0, currentDate=Tue Apr 08 20:35:00 CST 2025 2025-04-08 20:35:00.403 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:30:01.0, nextRunTime=2025-04-08 20:35:00.0} 2025-04-08 20:35:00.406 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 20:35:00.406 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:35:00.406+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:35:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,3dde99d6c182e21d,74a2875f805e25db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:00.496 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:30:01.0, nextRun=2025-04-08 20:35:00.0, currentDate=Tue Apr 08 20:35:00 CST 2025 2025-04-08 20:35:00.496 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:30:01.0, nextRunTime=2025-04-08 20:35:00.0} 2025-04-08 20:35:00.501 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 20:35:00.504 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 20:35:00.504 INFO 3009027 --- [task-coordinator1,1308668168866431,0a9ad19ca9d8c711] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 20:35:00.889 INFO 3009027 --- [subtask-coordinator3,1308668168866431,856e1b1e2e68e6ed] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 43, operation:EXECUTE, state: RUNNING 2025-04-08 20:35:00.963 WARN 3009027 --- [ocp-async-1,41075d53cf102243,45a1a45b3cc5a2e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:00.963 INFO 3009027 --- [ocp-async-9,a610fc946f6e5669,08d3a9e87d75dab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:00.985 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:35:00.987 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=43, context=Context{parallelIdx=-1, stringMap={task_instance_id=5, task_operation=execute, latest_execution_start_time=2025-04-08T20:35:00.887+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=43}, listMap={}}, executor=192.168.1.61 2025-04-08 20:35:00.991 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.993 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 20:35:00.993 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.993 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.994 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.994 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.994 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.995 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:00.995 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:35:01.053 INFO 3009027 --- [subtask-executor4,1308668168866431,b7e9dcfda18536d8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 43, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:35:01.201 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:5, status:SUCCESS 2025-04-08 20:35:01.201 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:5, status:SUCCESS 2025-04-08 20:35:02.151 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:02.151 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:02.898 INFO 3009027 --- [ocp-schedules-26,ecb4f5afe0a461f0,d06e3950254168c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:35:02.898 INFO 3009027 --- [ocp-schedules-6,efa140f579033b5d,704c017e28e76dfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:35:02.925 INFO 3009027 --- [ocp-async-9,f05146580f594130,43af2b2081079a83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:35:02.936 INFO 3009027 --- [ocp-schedules-2,288c4937d9cdd414,af5d0463a638818e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:35:02.937 INFO 3009027 --- [ocp-schedules-2,288c4937d9cdd414,af5d0463a638818e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:02.937 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:35:02.959 INFO 3009027 --- [ocp-schedules-19,4811dc0e0b1e91c3,29ee346fa79589ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:35:02.963 INFO 3009027 --- [ocp-async-6,d8533be72cf7d630,04f23c2fbc22c517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:02.963 WARN 3009027 --- [ocp-async-2,0bbc655c8493c943,b043c1b8ec30e5ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:35:03.176956+08:00, lastProcessTime=2025-04-08T20:34:53.177708+08:00 2025-04-08 20:35:03.307 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:35:03.176956+08:00 2025-04-08 20:35:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,379598a12fade196,7fafcb5bfb2d8d73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:04.670 INFO 3009027 --- [ocp-schedules-3,b1de70cd161b1f9c,17cdb69cc90108ae] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:35:04.677 INFO 3009027 --- [ocp-schedules-3,b1de70cd161b1f9c,17cdb69cc90108ae] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:35:04.677 INFO 3009027 --- [ocp-schedules-3,b1de70cd161b1f9c,17cdb69cc90108ae] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:35:04.963 WARN 3009027 --- [ocp-async-6,0e31796438323a72,55d43799ce875b33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:04.963 INFO 3009027 --- [ocp-async-10,7aa23a98f6d82eb5,9c6f24056f0d691e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:05.151 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:05.152 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:05.162 INFO 3009027 --- [ocp-schedules-14,cc8568fce58ec341,f7b45069f27de97c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:35:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,a59f5fae972d24be,fba381da27cd1aae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:06.809 INFO 3009027 --- [ocp-schedules-24,816c6ee353c33cf1,75813f191f10c76d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:35:06.810 INFO 3009027 --- [ocp-schedules-24,816c6ee353c33cf1,75813f191f10c76d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:35:06.963 WARN 3009027 --- [ocp-async-3,8a8647335adaa9c5,2ad4b163302b8886] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:06.963 INFO 3009027 --- [ocp-async-10,bbb0894387f20152,0ebf2259e20b7848] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:08.152 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:08.153 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:08.963 WARN 3009027 --- [ocp-async-2,6ef1e32ec4a74c8a,0ee3ef6837e5d98e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:08.963 INFO 3009027 --- [ocp-async-7,4f9ddd69a1ae88d1,f3f7b2b408a8becf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,a5addd378e2bfca4,0f33b2587ecd73ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:10.001 INFO 3009027 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:35:10.002 INFO 3009027 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:35:10.963 WARN 3009027 --- [ocp-async-6,4f17bf60e8d5f742,5a61594fc1e2f480] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:10.964 INFO 3009027 --- [ocp-async-1,a571f3f8accde8f7,e74690fc21549736] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:10.988 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:35:11.153 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:11.154 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,8100449252934516,5a4bd9288e690585] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:12.408 INFO 3009027 --- [prometheus-cache-monitor1,69b946ea4534daef,bc4f557f53b141e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1666,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:35:12.408 INFO 3009027 --- [prometheus-cache-monitor1,69b946ea4534daef,bc4f557f53b141e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:35:12.448 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T20:29:12.188, end=2025-04-08T20:34:12.36} 2025-04-08 20:35:12.449 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T20:29:12.203, end=2025-04-08T20:34:12.403} 2025-04-08 20:35:12.452 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 20:35:12.452 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:35:12.452 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 20:35:12.452 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:35:12.925 INFO 3009027 --- [ocp-async-1,29bb6c4c11df3fd8,203d43ae52637a6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:35:12.926 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:31:12.885, end=2025-04-08T20:33:12.907} 2025-04-08 20:35:12.928 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:35:12.928 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:35:12.937 INFO 3009027 --- [ocp-schedules-2,288c4937d9cdd414,af5d0463a638818e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:35:12.960 INFO 3009027 --- [ocp-schedules-27,618a23291dfa659e,c74f8f4579b684e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:35:12.964 WARN 3009027 --- [ocp-async-3,46817722f0d1f079,fc4bdb47f4b995d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:12.964 INFO 3009027 --- [ocp-async-9,22f8036f168f67f8,56999933c50a06d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:35:13.176994+08:00, lastProcessTime=2025-04-08T20:35:03.176956+08:00 2025-04-08 20:35:13.286 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:35:13.176994+08:00 2025-04-08 20:35:14.154 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:14.154 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:14.964 WARN 3009027 --- [ocp-async-10,6281b5e252a099fc,50171b184a73979f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:14.964 INFO 3009027 --- [ocp-async-5,aa5aa2d9663d5f59,1caa2c15ce5a2f2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:15.210 INFO 3009027 --- [ocp-schedules-20,c658aa25a2238511,584e318a0ae0221f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:35:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,94bf801d9bb4c1cd,a2418e8f5f1e17f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:15.609 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115640, nodeCount=418, affectRows=418 2025-04-08 20:35:15.612 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115640, nodeCount=415, validCount=415 2025-04-08 20:35:16.815 INFO 3009027 --- [ocp-schedules-26,93eb189df389c323,dab0f10a0bb6ad76] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:35:16.816 INFO 3009027 --- [ocp-schedules-26,93eb189df389c323,dab0f10a0bb6ad76] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:35:16.964 WARN 3009027 --- [ocp-async-7,3cbddd92b4dd6fe5,160b4608ad05407f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:16.964 INFO 3009027 --- [ocp-async-6,6a4e6d4ee4a3b5e2,3feb28c3e5298d75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:17.154 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:17.155 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,6304a931555b1022,4addf0e8cf8857cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:18.964 WARN 3009027 --- [ocp-async-3,2b25b86dda7f1b20,77af1f1fe432bed6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:18.964 INFO 3009027 --- [ocp-async-1,48f408d530f5b30e,d93fc7d9d38c13dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:20.155 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:20.156 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:20.964 WARN 3009027 --- [ocp-async-10,8dafe8c2371e453b,e1f8e056c0c6814f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:20.964 INFO 3009027 --- [ocp-async-1,a32d73807004a0b3,9a63c1646698bf78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,d64174c3d8377dd2,866508251bd9e56c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:22.895 INFO 3009027 --- [ocp-schedules-5,7eb63b188e3de8fd,0fee8d058b5222d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:35:22.896 INFO 3009027 --- [ocp-schedules-1,f27f0b4d5fdcaa65,b3c6529849b42b82] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:35:22.896 INFO 3009027 --- [ocp-schedules-11,ae88c123fa38fb9b,ec611db4759caf16] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:35:22.896 INFO 3009027 --- [ocp-schedules-29,ee4b71c514f015fd,a1e1a3f32672ee16] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:35:22.897 INFO 3009027 --- [ocp-schedules-11,ae88c123fa38fb9b,ec611db4759caf16] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:22.897 INFO 3009027 --- [ocp-schedules-21,688f7661ebf376ae,5620f5e292cd8e96] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:35:22.897 INFO 3009027 --- [ocp-schedules-21,688f7661ebf376ae,5620f5e292cd8e96] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:35:22.898 INFO 3009027 --- [ocp-schedules-8,174d592e2f5ae58f,b8059a2bc43ae486] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:35:22.898 INFO 3009027 --- [ocp-schedules-8,174d592e2f5ae58f,b8059a2bc43ae486] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:35:22.898 INFO 3009027 --- [ocp-schedules-3,419d6b7f47a3efa0,21caebf54dafd4c9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:35:22.898 INFO 3009027 --- [ocp-schedules-3,419d6b7f47a3efa0,21caebf54dafd4c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:35:22.899 INFO 3009027 --- [ocp-schedules-2,dd269a6acf94d9e2,7b8ea6eda1941688] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:35:22.899 INFO 3009027 --- [ocp-schedules-2,dd269a6acf94d9e2,7b8ea6eda1941688] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:35:22.902 INFO 3009027 --- [ocp-schedules-29,ee4b71c514f015fd,a1e1a3f32672ee16] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:35:22.920 INFO 3009027 --- [ocp-schedules-4,b1de38281e83f0bf,2ab7b4df9e9cc833] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 20:35:22.920 INFO 3009027 --- [ocp-schedules-4,b1de38281e83f0bf,2ab7b4df9e9cc833] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 20:35:22.925 INFO 3009027 --- [ocp-async-4,2ee75c3626e8fc87,4a7537d032cef7b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:35:22.937 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:35:22.937 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:35:22.938 INFO 3009027 --- [ocp-schedules-7,63a30e4e534dc3f9,42bd0db186696b78] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:35:22.938+08:00, clusters=[1] 2025-04-08 20:35:22.939 INFO 3009027 --- [ocp-schedules-7,63a30e4e534dc3f9,42bd0db186696b78] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:35:22.941 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:35:22.941 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:35:22.941 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:35:22.942 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:35:22.942 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:35:22.942 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:35:22.944 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:35:22.944 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:35:22.944 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:35:22.945 INFO 3009027 --- [ocp-schedules-22,c404bd2610a6084a,e13a4dcd73d38c81] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:35:22.952 INFO 3009027 --- [ocp-schedules-30,f7f42435ea7c89ba,9d5c6a27969b3223] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:35:22.952+08:00, clusters=[1] 2025-04-08 20:35:22.954 INFO 3009027 --- [ocp-schedules-30,f7f42435ea7c89ba,9d5c6a27969b3223] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:35:22.955 INFO 3009027 --- [ocp-schedules-28,f5f71af6e47c38b0,e124572806abceb8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:35:22.957 INFO 3009027 --- [ocp-schedules-12,76a4c8280c862295,87105e353b262884] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:35:22.957 INFO 3009027 --- [ocp-schedules-5,7eb63b188e3de8fd,0fee8d058b5222d7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:35:22.962 INFO 3009027 --- [ocp-schedules-1,12a085bfd22c1116,db566a2b4486710f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:35:22.964 WARN 3009027 --- [ocp-async-6,ef2074e61447315b,8316943c9397b4b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:22.964 INFO 3009027 --- [ocp-async-3,ec5bb23da068b836,0ad14d80cee8efd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:22.969 INFO 3009027 --- [ocp-schedules-8,3f44ff39d33e3cf6,76c94ab87788a315] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:35:22.981 INFO 3009027 --- [ocp-schedules-15,bef4fa7c7155a9fb,f3b6d3911e8e091d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:35:22.981 INFO 3009027 --- [ocp-schedules-15,bef4fa7c7155a9fb,f3b6d3911e8e091d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:35:22.984 INFO 3009027 --- [ocp-schedules-19,b02adedd75b86e8a,45aa0598e1e0968d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:35:22.984 INFO 3009027 --- [ocp-schedules-19,b02adedd75b86e8a,45aa0598e1e0968d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:35:22.988 INFO 3009027 --- [ocp-schedules-19,b02adedd75b86e8a,45aa0598e1e0968d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:35:22.988 INFO 3009027 --- [ocp-schedules-19,b02adedd75b86e8a,45aa0598e1e0968d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:35:22.991 INFO 3009027 --- [ocp-schedules-16,d1b136e5f440b676,538f57180564541c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:35:22.991 INFO 3009027 --- [ocp-schedules-16,b4d1e57d726577b7,1ac471476699afbf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:35:22.992 INFO 3009027 --- [ocp-schedules-16,b4d1e57d726577b7,1ac471476699afbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:22.997 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:35:22.997 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:35:22.998 INFO 3009027 --- [ocp-schedules-10,be498f16385ae10b,f09ce8f5a5705c98] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:35:23.000 INFO 3009027 --- [ocp-schedules-23,d379ff7145b41975,0028daaa7c27dc34] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:35:23.000 INFO 3009027 --- [ocp-schedules-23,d379ff7145b41975,0028daaa7c27dc34] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:35:23.003 INFO 3009027 --- [ocp-schedules-27,e5cec1f5a7af9538,3636b7edd4467c8f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:35:23.005 INFO 3009027 --- [ocp-schedules-27,e5cec1f5a7af9538,3636b7edd4467c8f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:35:23.005 INFO 3009027 --- [ocp-schedules-27,e5cec1f5a7af9538,3636b7edd4467c8f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:35:23.017 INFO 3009027 --- [ocp-schedules-9,bfa7f46b746d8501,560f9e7f54cd4d52] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:35:23.018 INFO 3009027 --- [ocp-schedules-9,bfa7f46b746d8501,560f9e7f54cd4d52] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:23.034 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:35:23.035 ERROR 3009027 --- [ocp-schedules-9,bfa7f46b746d8501,560f9e7f54cd4d52] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:35:23.035 INFO 3009027 --- [ocp-schedules-9,bfa7f46b746d8501,560f9e7f54cd4d52] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:35:23.069 INFO 3009027 --- [ocp-schedules-24,1188ce125697a210,a8d84f0af5d4a79f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:35:23.095 INFO 3009027 --- [ocp-schedules-20,c76cc8211830c9bc,fb1fae878831c86d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:35:23.095 INFO 3009027 --- [ocp-schedules-20,c76cc8211830c9bc,fb1fae878831c86d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:35:23.096 INFO 3009027 --- [ocp-schedules-20,c76cc8211830c9bc,fb1fae878831c86d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:35:23.096 INFO 3009027 --- [ocp-schedules-20,c76cc8211830c9bc,fb1fae878831c86d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766199115910868992 2025-04-08 20:35:23.096 INFO 3009027 --- [ocp-schedules-20,c76cc8211830c9bc,fb1fae878831c86d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:35:23.096 INFO 3009027 --- [ocp-schedules-20,c76cc8211830c9bc,fb1fae878831c86d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:35:23.156 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:23.157 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:35:23.175657+08:00, lastProcessTime=2025-04-08T20:35:13.176994+08:00 2025-04-08 20:35:23.257 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:35:23.175657+08:00 2025-04-08 20:35:23.804 INFO 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:35:23.804 INFO 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:35:23.807 INFO 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:35:23.808 WARN 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:35:23.809 INFO 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:35:23.810 WARN 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:35:23.812 INFO 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:35:23.814 WARN 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:35:23.815 ERROR 3009027 --- [ocp-schedules-5,e9d6a6875419a9c6,cd80a77c74af89e7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:35:23.908 INFO 3009027 --- [ocp-schedules-26,2dd4d74aed7a29bf,08455dfe3b4d45a9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:35:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,390516a8d009c56d,bd51c778142d4c22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:24.678 INFO 3009027 --- [ocp-schedules-17,d4c3a560c56a6418,5595d8ee0553b09d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:35:24.693 INFO 3009027 --- [ocp-schedules-17,d4c3a560c56a6418,5595d8ee0553b09d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:35:24.693 INFO 3009027 --- [ocp-schedules-17,d4c3a560c56a6418,5595d8ee0553b09d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:35:24.964 INFO 3009027 --- [ocp-async-9,af7e611c1e8f0c71,e1b7879a5c0a333b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:24.964 WARN 3009027 --- [ocp-async-5,f1d7a51230b7f035,73de683c013e317b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:25.231 INFO 3009027 --- [ocp-schedules-13,7e46d54bd7c236e4,0cb28c67d1d1e49b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:35:26.157 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:26.157 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:26.859 INFO 3009027 --- [ocp-schedules-30,6ffbc184a1e45b50,f1aa097142bb0572] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:35:26.861 INFO 3009027 --- [ocp-schedules-30,6ffbc184a1e45b50,f1aa097142bb0572] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:35:26.964 WARN 3009027 --- [ocp-async-9,d20e2e191a421b6b,77b8b69f21bc0bb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:26.965 INFO 3009027 --- [ocp-async-2,d8f2caa13fc0280e,4817ec606effb0cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:35:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,7259a528f98aebdd,78bab93f69460063] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:28.965 WARN 3009027 --- [ocp-async-8,c8f354479b37edea,7f0d777079869349] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:28.965 INFO 3009027 --- [ocp-async-6,37aaa137b45ff236,28ae3f28bb9ad065] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:29.157 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:29.158 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,f9c550cf96845cb5,c0e5ae013b333523] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:30.965 WARN 3009027 --- [ocp-async-5,d3c35d698a88e5aa,abd09a619eda9861] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:30.965 INFO 3009027 --- [ocp-async-6,5c553b7ca824ec8e,125ad806e59f9659] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:32.158 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:32.159 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:32.898 INFO 3009027 --- [ocp-schedules-21,688f7661ebf376ae,5620f5e292cd8e96] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:35:32.898 INFO 3009027 --- [ocp-schedules-11,ae88c123fa38fb9b,ec611db4759caf16] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:35:32.926 INFO 3009027 --- [ocp-async-1,8311d410218c557b,8bdb19e88f76e2c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:35:32.963 INFO 3009027 --- [ocp-schedules-1,356cb81765c29c0e,a2c6e3a2dfc22c37] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:35:32.965 INFO 3009027 --- [ocp-async-7,109d3e5b32170e40,e89d0482e4b30575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:32.965 WARN 3009027 --- [ocp-async-2,0c31fc49ec636cef,26c431b0136a1640] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:35:33.176783+08:00, lastProcessTime=2025-04-08T20:35:23.175657+08:00 2025-04-08 20:35:33.277 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:35:33.176783+08:00 2025-04-08 20:35:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,3603be52e7dee42a,c508a9929101d69d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:34.965 WARN 3009027 --- [ocp-async-7,a0be253534ce3148,4e0ee1a13533e5ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:34.965 INFO 3009027 --- [ocp-async-3,229d4793c39288fe,aa8a6b32c21109eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:35.159 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:35.160 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:35.267 INFO 3009027 --- [ocp-schedules-13,ffaa6b1bd030fe08,c0c1c9f7efd8605a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:35:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,d99f4389a847ea15,af2141278ea3e147] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:36.876 INFO 3009027 --- [ocp-schedules-14,4e7a9fec2dc83917,f922af73186d439e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:35:36.877 INFO 3009027 --- [ocp-schedules-14,4e7a9fec2dc83917,f922af73186d439e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:35:36.965 WARN 3009027 --- [ocp-async-10,cffab8e1ac87b3d4,58b9f9a46d4380b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:36.965 INFO 3009027 --- [ocp-async-9,73a17e11ec2826a7,4d58d1946280c787] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:38.160 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:38.160 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:38.965 WARN 3009027 --- [ocp-async-4,454a3f2fa4698cf3,ba107846c8d27395] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:38.965 INFO 3009027 --- [ocp-async-9,6ca50b29e8ce1695,5c50d69391e36562] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,4d49f00d65b74356,f2fd94e41be65ff0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:40.001 INFO 3009027 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:35:40.001 INFO 3009027 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:35:40.966 WARN 3009027 --- [ocp-async-1,41336d6240014424,69f232eef88f73e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:40.966 INFO 3009027 --- [ocp-async-7,8a136a0999837d01,70b539e6a26a9892] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:41.161 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:41.161 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,795ecac2e8ae0811,5b1bb1ec9045997e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:42.926 INFO 3009027 --- [ocp-async-7,cbbee65c2b41cda4,aa64615c165a2a17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:35:42.964 INFO 3009027 --- [ocp-schedules-1,19ffe977ae87d4a4,1d63cd5cbffe2269] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:35:42.966 WARN 3009027 --- [ocp-async-10,026effa2690d55da,93161b1e0bf0363f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:42.966 INFO 3009027 --- [ocp-async-2,e87c76cfccd97e7a,926b9cdd0e8ed491] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:35:43.176381+08:00, lastProcessTime=2025-04-08T20:35:33.176783+08:00 2025-04-08 20:35:43.281 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:35:43.176381+08:00 2025-04-08 20:35:44.161 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:44.162 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:44.694 INFO 3009027 --- [ocp-schedules-23,d3539746685672a7,90932fbdde52d651] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:35:44.709 INFO 3009027 --- [ocp-schedules-23,d3539746685672a7,90932fbdde52d651] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:35:44.709 INFO 3009027 --- [ocp-schedules-23,d3539746685672a7,90932fbdde52d651] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:35:44.966 WARN 3009027 --- [ocp-async-8,403c1fd11f6ad619,9838c86bd7a7bd3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:44.966 INFO 3009027 --- [ocp-async-9,cb1a88c86dbf1a00,8366af6b19265ca1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:45.289 INFO 3009027 --- [ocp-schedules-24,019693452ae371d5,8f30a2d76ad03bf6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:35:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,f28b2acee03d3543,1162aef5e0c08603] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:46.908 INFO 3009027 --- [ocp-schedules-10,40d0fd59881922c7,a1a19f416d155383] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:35:46.910 INFO 3009027 --- [ocp-schedules-10,40d0fd59881922c7,a1a19f416d155383] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:35:46.966 WARN 3009027 --- [ocp-async-9,9805577c51afb036,9ea0007f63dfeeae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:46.966 INFO 3009027 --- [ocp-async-5,a620bab008abfb37,30d3de523c3faf1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:47.162 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:47.163 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,b0a8d6e82a0b5968,5d6d14c0a11cde06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:48.966 WARN 3009027 --- [ocp-async-10,4e5f22445a1f93ae,3f49cdba83c50fe5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:48.966 INFO 3009027 --- [ocp-async-10,8cd6f347ec07442b,d23857dba437d91f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:50.163 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:50.164 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:50.966 WARN 3009027 --- [ocp-async-8,45618bbf556b195c,bca71a79a9b8db91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:50.966 INFO 3009027 --- [ocp-async-2,380c859df1e9a884,cef04a4a555dbfd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,aabe33215f0bb2e5,3e2484c05f05f378] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:52.896 INFO 3009027 --- [ocp-schedules-21,7475675b90fefebb,750334349944447f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:35:52.896 INFO 3009027 --- [ocp-schedules-8,13801b48e9a8a62f,2c9579cc85b319ef] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:35:52.896 INFO 3009027 --- [ocp-schedules-8,13801b48e9a8a62f,2c9579cc85b319ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:52.896 INFO 3009027 --- [ocp-schedules-28,58b204e6a1cefa84,0e601702e416ef24] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:35:52.897 INFO 3009027 --- [ocp-schedules-28,58b204e6a1cefa84,0e601702e416ef24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:35:52.898 INFO 3009027 --- [ocp-schedules-13,99d42625c3b5ebb3,3e513f17043e14e8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:35:52.898 INFO 3009027 --- [ocp-schedules-15,5834816ff999a6b0,486e1c413d5ecdd7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:35:52.898 INFO 3009027 --- [ocp-schedules-13,99d42625c3b5ebb3,3e513f17043e14e8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:35:52.898 INFO 3009027 --- [ocp-schedules-15,5834816ff999a6b0,486e1c413d5ecdd7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:35:52.899 INFO 3009027 --- [ocp-schedules-4,38e0e244abccbb4e,6e2771b4ccc37a52] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:35:52.899 INFO 3009027 --- [ocp-schedules-4,38e0e244abccbb4e,6e2771b4ccc37a52] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:35:52.926 INFO 3009027 --- [ocp-async-2,c5a14fe96f6fca1a,fc74feadb5991297] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:35:52.966 INFO 3009027 --- [ocp-schedules-30,2eb7d11d9fb43145,3c24b0cbe7fe9f2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:35:52.966 WARN 3009027 --- [ocp-async-4,8a6fa25ee53fbe56,f8cca8c502132860] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:52.967 INFO 3009027 --- [ocp-async-9,4a0c0c34a1a1fc82,e8b55d97dc7dad88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:52.970 INFO 3009027 --- [ocp-schedules-2,67c3b71a42d0bb54,e7d7abc94d63320b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:35:52.976 INFO 3009027 --- [ocp-schedules-29,6df195e8392ea451,d47c1a31d62e5ac4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:35:52.998 INFO 3009027 --- [ocp-schedules-16,b4d1e57d726577b7,1ac471476699afbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:35:52.998 INFO 3009027 --- [ocp-schedules-16,8538fb041a0da001,b614328ad51e749a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:35:52.999 INFO 3009027 --- [ocp-schedules-16,8538fb041a0da001,b614328ad51e749a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:35:53.004 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:35:53.004 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:35:53.164 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:53.164 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:35:53.177068+08:00, lastProcessTime=2025-04-08T20:35:43.176381+08:00 2025-04-08 20:35:53.299 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:35:53.177068+08:00 2025-04-08 20:35:53.624 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,09830ab8fb50c56f,a9eae93e95cc4d33] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=09830ab8fb50c56f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744115753620896,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 20:35:53.628 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,09830ab8fb50c56f,a9eae93e95cc4d33] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=09830ab8fb50c56f, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744115753620896,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=6 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 20:35:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,89880f33992648c0,4730243bdfdd683b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:54.967 WARN 3009027 --- [ocp-async-6,da709dbe7d285e9c,a5f6f5562dc5ce6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:54.967 INFO 3009027 --- [ocp-async-9,94f758ca34964895,aadfb5c5ae6b6331] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:55.319 INFO 3009027 --- [ocp-schedules-25,aa3a1befe73818cb,91aa21c37cc9e5a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:35:56.164 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:56.165 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:35:56.912 INFO 3009027 --- [ocp-schedules-4,7f70e7f030e5f6dd,c86a6ad487086d03] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:35:56.914 INFO 3009027 --- [ocp-schedules-4,7f70e7f030e5f6dd,c86a6ad487086d03] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:35:56.967 WARN 3009027 --- [ocp-async-1,18b1238acd37e093,8d6ed66928128a50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:56.967 INFO 3009027 --- [ocp-async-8,58a6f5d8375e7092,6bf80987fbfa816a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,c55c76c7b10d51b9,376ff6791a4eb454] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115760, interval=60 2025-04-08 20:35:58.967 INFO 3009027 --- [ocp-async-8,08a71470f497fd39,f9d815c6dd2fd4eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:35:58.967 WARN 3009027 --- [ocp-async-7,eac9cbdb8ee124e3,3e171e6de58da786] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:35:59.165 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:35:59.166 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,a77e687027c6aeab,da82be42a35366f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:00.967 WARN 3009027 --- [ocp-async-8,11df32040857832f,929da2887d167455] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:00.967 INFO 3009027 --- [ocp-async-6,0db0f0c50c8747f1,af6986daa294e203] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:02.166 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:02.166 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:02.897 INFO 3009027 --- [ocp-schedules-8,13801b48e9a8a62f,2c9579cc85b319ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:36:02.897 INFO 3009027 --- [ocp-schedules-28,58b204e6a1cefa84,0e601702e416ef24] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:36:02.926 INFO 3009027 --- [ocp-async-6,1f745eeaa580092d,f91c805c159a80d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:36:02.967 WARN 3009027 --- [ocp-async-9,9aa009332c3111b1,39da686914f6b0b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:02.967 INFO 3009027 --- [ocp-async-4,ec3539cfac2a2817,4fff0151ff06e416] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:02.968 INFO 3009027 --- [ocp-schedules-10,76f6d07b443c7be8,0d28205b34c806ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:36:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:36:03.176362+08:00, lastProcessTime=2025-04-08T20:35:53.177068+08:00 2025-04-08 20:36:03.260 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:36:03.176362+08:00 2025-04-08 20:36:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,320e277e7b595797,8062f0de92aff187] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:04.710 INFO 3009027 --- [ocp-schedules-30,54aa77bf077eeaed,4f7d43d97544486b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:36:04.721 INFO 3009027 --- [ocp-schedules-30,54aa77bf077eeaed,4f7d43d97544486b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:36:04.721 INFO 3009027 --- [ocp-schedules-30,54aa77bf077eeaed,4f7d43d97544486b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:36:04.967 WARN 3009027 --- [ocp-async-2,62580ca466e3448c,b596febfe38ce5d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:04.968 INFO 3009027 --- [ocp-async-5,ec75d1188595a6b3,ace1f7ce7c9e91a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:05.167 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:05.167 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:05.339 INFO 3009027 --- [ocp-schedules-9,2343c919ab856720,a7f4ba0e2ce1cbbb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:36:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,b771bce7be76bd14,9a07888b8728bc61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:06.968 WARN 3009027 --- [ocp-async-3,76f16beb21b4db7a,4e3eafb53e255194] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:06.968 INFO 3009027 --- [ocp-async-10,e6c8a83bf0c9307f,82f36df148a8f4b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:07.007 INFO 3009027 --- [ocp-schedules-10,42b33e3f294381d6,c2d6b1e105e411e0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:36:07.008 INFO 3009027 --- [ocp-schedules-10,42b33e3f294381d6,c2d6b1e105e411e0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:36:08.167 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:08.168 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:08.968 WARN 3009027 --- [ocp-async-10,2497ef9dc1fd7b18,12ba42ea250bd318] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:08.968 INFO 3009027 --- [ocp-async-7,4d837d9b348c0e0a,b3a842e5e6206340] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,15f6fcd1f994fdf4,1ed26439711ad22e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:10.001 INFO 3009027 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:36:10.002 INFO 3009027 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:36:10.968 WARN 3009027 --- [ocp-async-6,5818af6f4d7b453b,3afc4ec7e443aba5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:10.968 INFO 3009027 --- [ocp-async-8,854c28790e06e332,c4c67a27213f1ae0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:10.988 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:36:11.168 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:11.169 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,71203fa96f8638e5,6fb2ef9dbe7b4fdd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:12.408 INFO 3009027 --- [prometheus-cache-monitor1,6fa83d3635abaac2,9cfe82b1fd8fe9ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1668,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:36:12.408 INFO 3009027 --- [prometheus-cache-monitor1,6fa83d3635abaac2,9cfe82b1fd8fe9ca] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:36:12.926 INFO 3009027 --- [ocp-async-4,7e97847e918e3a8f,c094e4394da4fcb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:36:12.937 INFO 3009027 --- [ocp-schedules-24,e59ea637b97e73ac,5717f054352e985f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:36:12.938 INFO 3009027 --- [ocp-schedules-24,e59ea637b97e73ac,5717f054352e985f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:12.938 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:36:12.968 WARN 3009027 --- [ocp-async-9,dbda0c2ee017e860,7de89d24ae27c638] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:12.968 INFO 3009027 --- [ocp-async-5,dee8158f7945f2ea,f62b2af57515b5ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:12.969 INFO 3009027 --- [ocp-schedules-25,62df6d30da5b1d97,a7b9b4d5a88809f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:36:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:36:13.176902+08:00, lastProcessTime=2025-04-08T20:36:03.176362+08:00 2025-04-08 20:36:13.327 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:36:13.176902+08:00 2025-04-08 20:36:14.169 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:14.169 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:14.968 WARN 3009027 --- [ocp-async-5,c084f0547269b963,7a7c102248f21dd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:14.968 INFO 3009027 --- [ocp-async-7,72befe6130ff992f,b3318ad4490d4b60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:15.367 INFO 3009027 --- [ocp-schedules-10,05f7412b6d70fda5,badf663f0afff7ac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:36:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,bffe7aa9db05742f,4d9808a69933d340] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:15.778 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115700, nodeCount=418, affectRows=418 2025-04-08 20:36:15.781 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115700, nodeCount=415, validCount=415 2025-04-08 20:36:16.968 WARN 3009027 --- [ocp-async-2,720251e65b300e43,f0c9d0c989cd16f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:16.968 INFO 3009027 --- [ocp-async-8,72983ebe7310518a,f91e646799e0f33f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:17.028 INFO 3009027 --- [ocp-schedules-19,3d79932309e6ddb6,f8e1d6d8fafd0c69] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:36:17.029 INFO 3009027 --- [ocp-schedules-19,3d79932309e6ddb6,f8e1d6d8fafd0c69] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:36:17.169 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:17.170 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,caa5fcdf0041e537,a871cbd4f2e02d0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:18.969 WARN 3009027 --- [ocp-async-9,2f525a1fa6672c2e,b4d344e5451c9735] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:18.969 INFO 3009027 --- [ocp-async-3,6fc46351c011d8a1,a0f397b0ebf10e57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:20.170 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:20.171 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:20.969 WARN 3009027 --- [ocp-async-5,2d5821d8e7d34a84,e88e4728ecd7cd55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:20.969 INFO 3009027 --- [ocp-async-3,b393ab871541037f,9f4f1cf37b0e9bcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,748641cf9efc4a79,95cc975fa6ff4137] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:22.888 INFO 3009027 --- [ocp-schedules-30,479163059548b0a5,79600108debc7939] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T20:36:22.888+08:00, tenantSize=0 2025-04-08 20:36:22.895 INFO 3009027 --- [ocp-schedules-20,7c2e21af070971ce,f395028492de7f9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:36:22.896 INFO 3009027 --- [ocp-schedules-7,fd91ae84a3ad28c3,feae9b28dab66737] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:36:22.896 INFO 3009027 --- [ocp-schedules-1,47cc12d075d48a5f,8ffde852f1b3f4d3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:36:22.896 INFO 3009027 --- [ocp-schedules-4,5d1d48cc19b8bdf4,8686c96528d27e6e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:36:22.896 INFO 3009027 --- [ocp-schedules-3,48d1013889390588,537543fa744c09d0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:36:22.897 INFO 3009027 --- [ocp-schedules-1,47cc12d075d48a5f,8ffde852f1b3f4d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:22.897 INFO 3009027 --- [ocp-schedules-3,48d1013889390588,537543fa744c09d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:36:22.899 INFO 3009027 --- [ocp-schedules-23,83153cf348e04486,38962b19c162b96e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:36:22.899 INFO 3009027 --- [ocp-schedules-9,b2ef321db84eb881,7262b266bda96621] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:36:22.899 INFO 3009027 --- [ocp-schedules-9,b2ef321db84eb881,7262b266bda96621] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:36:22.899 INFO 3009027 --- [ocp-schedules-23,83153cf348e04486,38962b19c162b96e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:36:22.899 INFO 3009027 --- [ocp-schedules-14,9f0505d7ed8f1ea3,ffc244fa12cf4ed8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:36:22.899 INFO 3009027 --- [ocp-schedules-14,9f0505d7ed8f1ea3,ffc244fa12cf4ed8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:36:22.903 INFO 3009027 --- [ocp-schedules-4,5d1d48cc19b8bdf4,8686c96528d27e6e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:36:22.913 INFO 3009027 --- [ocp-schedules-26,b7fbeeb59cfdb45b,67f9b8b052e98dbe] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1670,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:36:22.913 INFO 3009027 --- [ocp-schedules-26,b7fbeeb59cfdb45b,67f9b8b052e98dbe] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:36:22.926 INFO 3009027 --- [ocp-async-2,9455fd5f9e132e33,81427b3b1c619ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:36:22.928 INFO 3009027 --- [ocp-schedules-27,04c450f9c419512e,32747d889bc0694a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:36:22.928 INFO 3009027 --- [ocp-schedules-20,7c2e21af070971ce,f395028492de7f9d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:36:22.933 INFO 3009027 --- [ocp-schedules-22,e1435b2a4b3b626e,77cbb0bc6233e470] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 20:36:22.933 INFO 3009027 --- [ocp-schedules-22,e1435b2a4b3b626e,77cbb0bc6233e470] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 20:36:22.939 INFO 3009027 --- [ocp-schedules-24,e59ea637b97e73ac,5717f054352e985f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:36:22.944 INFO 3009027 --- [ocp-schedules-28,8c6f460c03194b73,6cc8c89d44679252] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:36:22.944+08:00, clusters=[1] 2025-04-08 20:36:22.945 INFO 3009027 --- [ocp-schedules-28,8c6f460c03194b73,6cc8c89d44679252] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:36:22.956 INFO 3009027 --- [ocp-schedules-2,5809255c7d8401ef,5ed03d5546d45418] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:36:22.959 INFO 3009027 --- [ocp-schedules-30,823598334d7aa31a,da2657472260d15a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:36:22.959+08:00, clusters=[1] 2025-04-08 20:36:22.960 INFO 3009027 --- [ocp-schedules-30,823598334d7aa31a,da2657472260d15a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:36:22.969 WARN 3009027 --- [ocp-async-10,45a18514b3cd6961,6be9217b6665639a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:22.969 INFO 3009027 --- [ocp-async-8,321652028960926b,7282d35bc9b46715] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:22.970 INFO 3009027 --- [ocp-schedules-23,2d620e648333c8cf,4f2fbd164ca94d76] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:36:22.972 INFO 3009027 --- [ocp-schedules-14,957e57f49bde00e5,ed472d0c36876373] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:36:22.972 INFO 3009027 --- [ocp-schedules-4,a9ca81ed6c986fc9,44c70ac01807f123] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:36:22.973 INFO 3009027 --- [ocp-schedules-4,a9ca81ed6c986fc9,44c70ac01807f123] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:22.982 INFO 3009027 --- [ocp-schedules-26,53549253ac117b81,f3c5693a013b8ca1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:36:22.982 INFO 3009027 --- [ocp-schedules-26,53549253ac117b81,f3c5693a013b8ca1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:36:22.984 ERROR 3009027 --- [ocp-schedules-4,a9ca81ed6c986fc9,44c70ac01807f123] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:36:22.984 INFO 3009027 --- [ocp-schedules-4,a9ca81ed6c986fc9,44c70ac01807f123] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:36:22.988 INFO 3009027 --- [ocp-schedules-13,24d99f94a9b4a5ba,7a8c870593a0b9ac] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:36:22.988 INFO 3009027 --- [ocp-schedules-13,24d99f94a9b4a5ba,7a8c870593a0b9ac] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:36:22.992 INFO 3009027 --- [ocp-schedules-13,24d99f94a9b4a5ba,7a8c870593a0b9ac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:36:22.992 INFO 3009027 --- [ocp-schedules-13,24d99f94a9b4a5ba,7a8c870593a0b9ac] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:36:22.998 INFO 3009027 --- [ocp-schedules-15,cb9e0da34cdceaf5,f5fc9a07f689c93f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:36:22.999 INFO 3009027 --- [ocp-schedules-15,cb9e0da34cdceaf5,f5fc9a07f689c93f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:23.002 INFO 3009027 --- [ocp-schedules-19,9916e6e6e751de65,21ec551c0114bd27] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:36:23.002 INFO 3009027 --- [ocp-schedules-19,9916e6e6e751de65,21ec551c0114bd27] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:36:23.005 INFO 3009027 --- [ocp-schedules-16,8538fb041a0da001,b614328ad51e749a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:36:23.005 INFO 3009027 --- [ocp-schedules-16,f14c94db43d723ce,c6a2a5c0c5aa13a2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:36:23.005 INFO 3009027 --- [ocp-schedules-29,3721cd5c881dff8a,cc2464bd36f08a4c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:36:23.005 INFO 3009027 --- [ocp-schedules-16,f14c94db43d723ce,c6a2a5c0c5aa13a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:23.006 INFO 3009027 --- [ocp-schedules-29,3721cd5c881dff8a,cc2464bd36f08a4c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:36:23.006 INFO 3009027 --- [ocp-schedules-29,3721cd5c881dff8a,cc2464bd36f08a4c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:36:23.008 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:23.008 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:36:23.010 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:36:23.010 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:36:23.035 INFO 3009027 --- [ocp-schedules-10,d2f26e04a1ab9e07,02d2d478dbb83e6e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:36:23.036 INFO 3009027 --- [ocp-schedules-10,d2f26e04a1ab9e07,02d2d478dbb83e6e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:23.052 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:36:23.052 ERROR 3009027 --- [ocp-schedules-10,d2f26e04a1ab9e07,02d2d478dbb83e6e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:36:23.053 INFO 3009027 --- [ocp-schedules-10,d2f26e04a1ab9e07,02d2d478dbb83e6e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:36:23.075 INFO 3009027 --- [ocp-schedules-27,0bc46d3c63c4727f,a872918d685abfe7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:36:23.096 INFO 3009027 --- [ocp-schedules-12,3ddbf5b9261ad042,c63a783164073fe2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:36:23.096 INFO 3009027 --- [ocp-schedules-12,3ddbf5b9261ad042,c63a783164073fe2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:36:23.097 INFO 3009027 --- [ocp-schedules-12,3ddbf5b9261ad042,c63a783164073fe2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:36:23.097 INFO 3009027 --- [ocp-schedules-12,3ddbf5b9261ad042,c63a783164073fe2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766199115910868992 2025-04-08 20:36:23.097 INFO 3009027 --- [ocp-schedules-12,3ddbf5b9261ad042,c63a783164073fe2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:36:23.097 INFO 3009027 --- [ocp-schedules-12,3ddbf5b9261ad042,c63a783164073fe2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:36:23.171 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:23.172 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:36:23.176775+08:00, lastProcessTime=2025-04-08T20:36:13.176902+08:00 2025-04-08 20:36:23.304 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:36:23.176775+08:00 2025-04-08 20:36:23.818 INFO 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:36:23.818 INFO 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:36:23.821 INFO 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:36:23.822 WARN 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:36:23.822 INFO 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:36:23.823 WARN 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:36:23.826 INFO 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:36:23.828 WARN 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:36:23.828 ERROR 3009027 --- [ocp-schedules-4,d5a66c52bded19e8,90787dd4b1bc9772] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:36:23.928 INFO 3009027 --- [ocp-schedules-25,5e182606307f1551,4dd23ec441769ab3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:36:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,f4f5bdf7642678c9,554a61da6a912ab7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:24.722 INFO 3009027 --- [ocp-schedules-20,7470767fec2f4bfa,93797b409550063e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:36:24.729 INFO 3009027 --- [ocp-schedules-20,7470767fec2f4bfa,93797b409550063e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:36:24.729 INFO 3009027 --- [ocp-schedules-20,7470767fec2f4bfa,93797b409550063e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:36:24.969 WARN 3009027 --- [ocp-async-1,5869d0ef10974ab2,179727a670513d02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:24.969 INFO 3009027 --- [ocp-async-7,065fc6d84e8e148f,d40deb39bd4761b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:25.391 INFO 3009027 --- [ocp-schedules-25,9c0d1edab8cb7b29,0f22b0f7ef7574c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:36:26.172 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:26.172 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:26.969 WARN 3009027 --- [ocp-async-4,dac71913898231a0,fc9ba2cb2f5190a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:26.969 INFO 3009027 --- [ocp-async-7,17162cb06e6baec2,4f4f7bf49832fe9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:36:27.111 INFO 3009027 --- [ocp-schedules-29,de09f442d7d0339e,5abdedaf884d5340] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:36:27.112 INFO 3009027 --- [ocp-schedules-29,de09f442d7d0339e,5abdedaf884d5340] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:36:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,61d509f7033c3ac8,861531770e33d059] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:28.138 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:36:28.138 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:36:28.142 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:36:28.142 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:36:28.142 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:36:28.142 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:36:28.147 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:36:28.147 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:36:28.147 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:36:28.147 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:36:28.171 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:36:28.176 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:36:28.176 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:36:28.176 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:36:28.176 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:36:28.176 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:36:28.181 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:36:28.502 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:36:28.826 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.678876448S 2025-04-08 20:36:28.826 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:36:28.881 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:36:28.882 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:36:28.906 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:36:28.953 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:36:28.953 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:36:28.953 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:36:28.953 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:36:28.953 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:36:28.953 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:36:28.960 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:36:28.969 WARN 3009027 --- [ocp-async-10,fb9d518bcfb11430,9e542485dd658912] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:28.969 INFO 3009027 --- [ocp-async-9,f921fb374e74eece,e5f1973e9a773c57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:29.065 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] o.s.boot.SpringApplication : Started application in 0.922 seconds (JVM running for 830.682) 2025-04-08 20:36:29.065 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.922811383S 2025-04-08 20:36:29.173 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:29.173 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:29.348 INFO 3009027 --- [ocp-schedules-10,49f7a5f59eb336ba,23f73c2e38e145e3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:36:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,919796d4e0a53597,7cdbfc02ed9522e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:30.969 WARN 3009027 --- [ocp-async-8,d9ff59f57e6d42d9,09783e6f2ef8c107] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:30.969 INFO 3009027 --- [ocp-async-1,116f1b383d8eb4ed,116da0674388e214] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:32.173 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:32.174 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:32.897 INFO 3009027 --- [ocp-schedules-1,47cc12d075d48a5f,8ffde852f1b3f4d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:36:32.898 INFO 3009027 --- [ocp-schedules-3,48d1013889390588,537543fa744c09d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:36:32.917 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 20:36:32.917 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 20:36:32.922 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.922 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:36:32.923 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.923 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.923 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.924 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 20:36:32.927 INFO 3009027 --- [ocp-async-9,da0ad8f206d772e8,e67927c2c06f51b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:36:32.927 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.927 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.928 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.928 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 20:36:32.931 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 20:36:32.931 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 20:36:32.932 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.932 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:36:32.932 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.933 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.933 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.933 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 20:36:32.945 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.945 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.945 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.946 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 20:36:32.956 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.956 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.956 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.956 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 20:36:32.967 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.968 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.968 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.968 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 20:36:32.970 WARN 3009027 --- [ocp-async-2,1bd3af4535a72d91,a720ab4bff58a420] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:32.970 INFO 3009027 --- [ocp-async-6,26efb30b90ac913d,01f10a37c1a5f5f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:32.971 INFO 3009027 --- [ocp-schedules-9,97671e8d8e47de97,d29fc8c7eb08c5c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:36:32.980 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.980 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.981 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.981 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 20:36:32.986 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.986 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.986 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.986 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 20:36:32.990 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.990 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.990 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.991 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 20:36:32.995 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.995 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:32.995 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:32.996 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 20:36:33.000 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.000 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.001 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.001 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 20:36:33.005 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.005 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.005 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.006 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 20:36:33.009 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.010 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.010 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.010 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 20:36:33.017 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.017 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.017 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.017 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 20:36:33.028 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.028 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.029 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.029 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 20:36:33.032 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.032 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.033 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.033 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 20:36:33.038 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.038 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.039 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.039 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 20:36:33.043 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.043 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.044 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.044 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 20:36:33.047 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.047 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.047 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.047 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 20:36:33.050 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.051 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.051 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.051 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 20:36:33.054 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.054 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.055 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.055 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 20:36:33.058 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.059 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.059 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.059 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 20:36:33.063 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.063 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.063 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.064 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 20:36:33.067 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.067 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.068 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.068 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 20:36:33.071 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.071 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.072 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.072 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 20:36:33.075 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.075 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.076 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.076 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 20:36:33.079 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.079 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.079 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.079 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 20:36:33.082 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.082 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.083 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.083 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 20:36:33.088 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.088 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.088 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.088 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 20:36:33.092 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.092 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.092 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.092 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 20:36:33.095 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.096 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.096 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.096 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 20:36:33.099 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.100 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.100 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.100 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 20:36:33.103 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.103 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.103 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.104 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 20:36:33.107 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.107 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.107 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.108 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 20:36:33.111 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.111 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.112 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.112 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 20:36:33.115 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.115 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.115 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.116 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 20:36:33.121 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.121 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.122 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.122 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 20:36:33.127 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.127 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.127 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.128 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 20:36:33.132 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.132 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.132 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.133 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 20:36:33.137 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.137 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.138 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.138 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 20:36:33.141 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.141 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.141 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.142 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 20:36:33.145 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.145 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.145 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.146 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 20:36:33.149 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.149 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.149 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.149 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 20:36:33.152 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.153 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.153 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.153 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 20:36:33.160 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.161 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.161 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.161 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 20:36:33.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:36:33.175981+08:00, lastProcessTime=2025-04-08T20:36:23.176775+08:00 2025-04-08 20:36:33.179 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.180 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.180 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.180 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 20:36:33.185 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.185 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.185 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.185 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 20:36:33.190 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.191 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.191 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.191 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 20:36:33.196 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.196 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.196 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.196 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 20:36:33.199 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.199 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.200 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.200 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 20:36:33.203 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.203 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:36:33.203 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:33.203 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 20:36:33.207 INFO 3009027 --- [ocp-schedules-26,b59c7f32299000cb,6697c0dc2e160493] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 20:36:33.316 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:36:33.175981+08:00 2025-04-08 20:36:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,3c03010e887a1cba,5947cf317a86f97d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:34.970 WARN 3009027 --- [ocp-async-6,2b4813b44c57e9d2,8e26f54dbe96723b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:34.970 INFO 3009027 --- [ocp-async-8,eab4e5c3c66305bc,39167bbb3ef379ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:35.174 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:35.175 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:35.421 INFO 3009027 --- [ocp-schedules-9,a1f14a731a909374,e470461afb828182] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:36:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,eab921b44997655e,9a43ea7da64bfaab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:36.970 WARN 3009027 --- [ocp-async-8,309f9b36672dd72b,024c300b02f2387f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:36.970 INFO 3009027 --- [ocp-async-4,50f40c03ecfed0c2,ccfe54cc1a87e203] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:37.115 INFO 3009027 --- [ocp-schedules-6,6c2b5c02145cb8b9,5c93b119c9526c65] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:36:37.116 INFO 3009027 --- [ocp-schedules-6,6c2b5c02145cb8b9,5c93b119c9526c65] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:36:38.175 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:38.175 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:38.970 INFO 3009027 --- [ocp-async-2,8e614f7436081a79,a71fe51db5660f36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:38.970 WARN 3009027 --- [ocp-async-10,8a11c14159932b81,293d456633217b55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,1a99b086ba318d72,9976dd35d29547fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:40.001 INFO 3009027 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:36:40.001 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:36:40.970 WARN 3009027 --- [ocp-async-2,86876474854bac00,08744017ff4a2e27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:40.970 INFO 3009027 --- [ocp-async-6,5e5cc6bf16d30539,d76f6dd23daa0ca4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:40.970 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:36:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:36:40.972 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:36:40.972 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:36:40.976 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1), status=TAKINGOVER 2025-04-08 20:36:40.976 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 20:36:41.175 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:41.176 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,b1a879319caffc99,5c9dd90672dfa701] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:42.927 INFO 3009027 --- [ocp-async-6,bdb4fe269086de7c,e77d707451c99a1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:36:42.970 WARN 3009027 --- [ocp-async-9,f2f7b4650e3528fb,bbf0050ae7d69f2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:42.970 INFO 3009027 --- [ocp-async-8,81a8b720e460fabe,45d4643bee3d8823] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:42.972 INFO 3009027 --- [ocp-schedules-28,4d46845f917357c3,84234e20ca1e574f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:36:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:36:43.176729+08:00, lastProcessTime=2025-04-08T20:36:33.175981+08:00 2025-04-08 20:36:43.301 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:36:43.176729+08:00 2025-04-08 20:36:44.176 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:44.177 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:44.730 INFO 3009027 --- [ocp-schedules-24,de8015da54fed54b,34e60028a51f43d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:36:44.799 INFO 3009027 --- [ocp-schedules-24,de8015da54fed54b,34e60028a51f43d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:36:44.799 INFO 3009027 --- [ocp-schedules-24,de8015da54fed54b,34e60028a51f43d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:36:44.970 WARN 3009027 --- [ocp-async-1,23d3ff324f5e4c3b,9e5eecd8ae46a37c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:44.970 INFO 3009027 --- [ocp-async-8,185e289283a6e337,b3bd6b6aee4733b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,a3bb2a9f3585f008,70c67fea2f025136] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:45.466 INFO 3009027 --- [ocp-schedules-22,7bbf1bf025dcbdd2,c307295cf7c40a1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:36:46.970 WARN 3009027 --- [ocp-async-1,4cd360c378f58a26,21891546b7fa26cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:46.971 INFO 3009027 --- [ocp-async-5,5b89a8fa308d9df3,5d92d71709a3a8f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:47.149 INFO 3009027 --- [ocp-schedules-11,7c2315fd851c5eab,ed8718c01bf697fb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:36:47.150 INFO 3009027 --- [ocp-schedules-11,7c2315fd851c5eab,ed8718c01bf697fb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:36:47.177 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:47.177 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,3f3e98c8e8fe3720,ecf6530e3aaa45b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:48.971 WARN 3009027 --- [ocp-async-5,29763cfcc868078c,f226ace5a3dbc9b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:48.971 INFO 3009027 --- [ocp-async-8,d5f9e35035cd3676,b5d3b00b8b1d41b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:50.178 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:50.178 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:50.531 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,d6a241cecd9b2fb4,b511be0c251ab415] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/retry, client 10.11.12.102, traceId d6a241cecd9b2fb4] 2025-04-08 20:36:50.536 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,d6a241cecd9b2fb4,b511be0c251ab415] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/retry?taskInstanceId=2, client=10.11.12.102, traceId=d6a241cecd9b2fb4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=2,] 2025-04-08 20:36:50.585 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,d6a241cecd9b2fb4,b511be0c251ab415] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 26, current state: FAILED, new state: READY 2025-04-08 20:36:50.598 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,d6a241cecd9b2fb4,b511be0c251ab415] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/retry client=10.11.12.102, traceId=d6a241cecd9b2fb4, duration=70 ms] 2025-04-08 20:36:50.598 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,d6a241cecd9b2fb4,b511be0c251ab415] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=2, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=null, startTime=2025-04-08T20:36:50.536+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 20:36:50.781 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,be439694942e2589,3d9ad904466da02e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=be439694942e2589, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:36:50.790 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,be439694942e2589,3d9ad904466da02e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=be439694942e2589, duration=13 ms] 2025-04-08 20:36:50.971 WARN 3009027 --- [ocp-async-10,6d209b3491bf907c,3c298eac76f496b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:50.971 INFO 3009027 --- [ocp-async-4,0e94b848822f4698,5cb516207aa1ee54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:51.095 INFO 3009027 --- [subtask-coordinator3,1308668168866431,6f0d698b2169aadc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:RETRY, state: RUNNING 2025-04-08 20:36:51.184 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:36:51.266 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=26, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:36:51.089+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=26}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:36:51.282 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 20:36:51.286 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:36:51.287 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:36:51.295 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:51.313 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.355 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:51.363 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1df8b1179b57425f,969532d7ac9ef8e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=1df8b1179b57425f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:36:51.366 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1df8b1179b57425f,969532d7ac9ef8e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=1df8b1179b57425f, duration=6 ms] 2025-04-08 20:36:51.374 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.376 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:36:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,350e1f1c7d35f3c3,ae6df90120a2760b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:51.402 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.425 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.443 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:51.464 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.524 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:36:51.675 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.678 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.716 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:51.740 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.743 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:36:51.769 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.771 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.776 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 20:36:51.824 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:51.841 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:51.846 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:36:52.255 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,b6760c776d93e92f,cbf35112ee983cf4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=b6760c776d93e92f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:36:52.265 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,b6760c776d93e92f,cbf35112ee983cf4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=b6760c776d93e92f, duration=13 ms] 2025-04-08 20:36:52.220 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:52.302 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:52.376 WARN 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:36:52.425 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:36:52.537 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,85ea9cd6335af895,aab62abf13a27b2b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=85ea9cd6335af895, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:36:52.494 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 20:36:52.586 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,85ea9cd6335af895,aab62abf13a27b2b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=85ea9cd6335af895, duration=53 ms] 2025-04-08 20:36:52.584 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:52.615 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:52.620 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:36:52.896 INFO 3009027 --- [ocp-schedules-7,b5195a3d4117c8fd,8ddf87b8610f75b4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:36:52.896 INFO 3009027 --- [ocp-schedules-27,4721d7ed490259fe,f60b9b0468f920b2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:36:52.896 INFO 3009027 --- [ocp-schedules-12,995574cd9e3d527f,3ea20a052d43f3a1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:36:52.897 INFO 3009027 --- [ocp-schedules-27,4721d7ed490259fe,f60b9b0468f920b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:52.897 INFO 3009027 --- [ocp-schedules-12,995574cd9e3d527f,3ea20a052d43f3a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:36:52.898 INFO 3009027 --- [ocp-schedules-20,e44ade30c5d58ab4,524e9d1f893cdcda] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:36:52.898 INFO 3009027 --- [ocp-schedules-1,f46c13ff186c6319,cbabfb811e07b030] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:36:52.898 INFO 3009027 --- [ocp-schedules-20,e44ade30c5d58ab4,524e9d1f893cdcda] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:36:52.898 INFO 3009027 --- [ocp-schedules-1,f46c13ff186c6319,cbabfb811e07b030] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:36:52.899 INFO 3009027 --- [ocp-schedules-29,3897775adb176df3,b45d0cfedf29dafa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:36:52.899 INFO 3009027 --- [ocp-schedules-29,3897775adb176df3,b45d0cfedf29dafa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:36:52.927 INFO 3009027 --- [ocp-async-4,5b36b519ec9bfd74,cdf9aba93797616a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:36:52.958 INFO 3009027 --- [ocp-schedules-5,73aa798c0ae9948e,1c446d60b8e0f094] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:36:52.971 WARN 3009027 --- [ocp-async-7,49fdb7c6b86231c4,74444c7ba8ec2d6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:52.971 INFO 3009027 --- [ocp-async-6,e5fe1ac5c2cd6376,b09fb218950bb1cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:52.973 INFO 3009027 --- [ocp-schedules-6,89cb3a6a12e9ce9d,c25e04622a745d9e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:36:52.973 INFO 3009027 --- [ocp-schedules-25,3ece7be5978963a8,87aa3aaa377bb905] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:36:52.999 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.010 INFO 3009027 --- [ocp-schedules-16,f14c94db43d723ce,c6a2a5c0c5aa13a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:36:53.010 INFO 3009027 --- [ocp-schedules-16,1572921f19d62f04,4aedd02a05ea0369] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:36:53.011 INFO 3009027 --- [ocp-schedules-16,1572921f19d62f04,4aedd02a05ea0369] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:36:53.016 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:36:53.016 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:36:53.002 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.019 WARN 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 20:36:53.033 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 20:36:53.048 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 20:36:53.052 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 20:36:53.058 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:36:53.062 WARN 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:36:53.069 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:36:53.072 WARN 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:36:53.076 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:36:53.077 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:36:53.084 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:53.141 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:36:53.175747+08:00, lastProcessTime=2025-04-08T20:36:43.176729+08:00 2025-04-08 20:36:53.178 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:53.179 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:53.153 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:53.213 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.253 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:36:53.380 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:36:53.175747+08:00 2025-04-08 20:36:53.336 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.381 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.426 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:53.466 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.506 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:36:53.542 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,9498986f9360ac5d,68be6799e495a174] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=9498986f9360ac5d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:36:53.551 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,9498986f9360ac5d,68be6799e495a174] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=9498986f9360ac5d, duration=12 ms] 2025-04-08 20:36:53.565 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.568 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.578 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 20:36:53.596 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:36:53.613 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:36:53.638 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,07fe9f07c0469c57,a186bf88a6d0b2f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=07fe9f07c0469c57, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:36:53.642 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,07fe9f07c0469c57,a186bf88a6d0b2f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=07fe9f07c0469c57, duration=8 ms] 2025-04-08 20:36:53.654 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 20:36:53.688 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:36:53.766 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:36:53.773 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 20:36:53.825 ERROR 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 20:36:53.838 ERROR 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:36:53.852 WARN 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:36:53.858 INFO 3009027 --- [manual-subtask-executor14,73ac1c39155401d0,802410c93de382b5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:RETRY, state: FAILED 2025-04-08 20:36:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,ecb4786846fe6be2,7973995a9019ef48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:54.510 INFO 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=2, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, createTime=2025-04-08T20:29:48+08:00, startTime=2025-04-08T20:29:48.360+08:00, endTime=2025-04-08T20:33:41.137+08:00, subtasks=[SubtaskInstanceEntity{id=31, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=12, startTime=null, endTime=null}, SubtaskInstanceEntity{id=12, name=Attach cluster attributes, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.AttachClusterAttributesTask, seriesId=14, startTime=null, endTime=null}, SubtaskInstanceEntity{id=2, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=18, name=Update host arch info, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateHostSystemInfoTask, seriesId=7, startTime=2025-04-08T20:29:52.261+08:00, endTime=2025-04-08T20:29:52.748+08:00}, SubtaskInstanceEntity{id=3, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=16, startTime=2025-04-08T20:30:05.536+0 2025-04-08 20:36:54.520 WARN 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:36:54.520 INFO 3009027 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:2, status:FAILED 2025-04-08 20:36:54.854 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,90863a7791263a3f,bbf0c48910f0b4cb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=90863a7791263a3f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:36:54.863 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,90863a7791263a3f,bbf0c48910f0b4cb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=90863a7791263a3f, duration=12 ms] 2025-04-08 20:36:54.955 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,088a21e7725ae70d,1fc0845ae93fe4ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=088a21e7725ae70d, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:36:54.960 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,088a21e7725ae70d,1fc0845ae93fe4ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=088a21e7725ae70d, duration=9 ms] 2025-04-08 20:36:54.971 WARN 3009027 --- [ocp-async-5,2d6c325ead967bb0,2aab96008477d5c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:54.971 INFO 3009027 --- [ocp-async-8,b120a20ef42793c7,a267f38a90e0cee5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:55.172 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,9f90339d9791ee52,4456dcae105ad4ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=9f90339d9791ee52, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:36:55.175 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,9f90339d9791ee52,4456dcae105ad4ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=9f90339d9791ee52, duration=6 ms] 2025-04-08 20:36:55.494 INFO 3009027 --- [ocp-schedules-14,da51b919adcee626,4f14a02f66031e8c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:36:56.179 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:56.179 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:36:56.971 WARN 3009027 --- [ocp-async-10,01c6b43d04902921,7fdb93eb34475ec7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:56.971 INFO 3009027 --- [ocp-async-3,0c2c1aa67d082cc8,0ace472d4f74e8b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:57.153 INFO 3009027 --- [ocp-schedules-7,a48e0fa282fce9be,442f9e50774dd8b4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:36:57.154 INFO 3009027 --- [ocp-schedules-7,a48e0fa282fce9be,442f9e50774dd8b4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:36:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,bac571b3bb6f29a4,e19415d3357ca3f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115820, interval=60 2025-04-08 20:36:58.971 WARN 3009027 --- [ocp-async-1,21aa345544b9b89b,ccc4f699c7a97640] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:36:58.971 INFO 3009027 --- [ocp-async-3,09d5651bfb8795b3,58be8a1d8ab3d9e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:36:59.180 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:36:59.180 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,b54319478a6428de,48d0f87643d8e0c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:00.972 INFO 3009027 --- [ocp-async-5,0595e5a19af72a38,327662b877b3e4d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:00.972 WARN 3009027 --- [ocp-async-6,d66e8e17a94a57ab,8adc10e334940fd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:02.180 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:02.181 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:02.897 INFO 3009027 --- [ocp-schedules-27,4721d7ed490259fe,f60b9b0468f920b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:37:02.898 INFO 3009027 --- [ocp-schedules-12,995574cd9e3d527f,3ea20a052d43f3a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:37:02.927 INFO 3009027 --- [ocp-async-5,811050d72c0ca9cd,ba63f8e0803eb961] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:37:02.972 WARN 3009027 --- [ocp-async-7,5597c97dd0a29fa2,c1f7e5ca36154bc1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:02.972 INFO 3009027 --- [ocp-async-2,0dfdb79da6800811,8e3b9226256b2870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:02.974 INFO 3009027 --- [ocp-schedules-20,a3bdd8c8a65608fb,9d5314aafb8807d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:37:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:37:03.177350+08:00, lastProcessTime=2025-04-08T20:36:53.175747+08:00 2025-04-08 20:37:03.261 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:37:03.177350+08:00 2025-04-08 20:37:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,662ab845ef56e9a0,9ddfdd742db2e812] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:04.801 INFO 3009027 --- [ocp-schedules-24,be3a23805387dbe8,65bf3cd8c3e5ae9b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:37:04.914 INFO 3009027 --- [ocp-schedules-24,be3a23805387dbe8,65bf3cd8c3e5ae9b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:37:04.914 INFO 3009027 --- [ocp-schedules-24,be3a23805387dbe8,65bf3cd8c3e5ae9b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:37:04.972 WARN 3009027 --- [ocp-async-4,cf3a898f9b766eb6,fb314da5571d3c83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:04.972 INFO 3009027 --- [ocp-async-2,fc3534fcf15107dd,0cb31f4051f2feb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:05.181 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:05.182 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:05.517 INFO 3009027 --- [ocp-schedules-23,97d2c9128377eaf4,453197ef0190be43] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:37:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,c378788fdb3434c9,3e68e911c8458970] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:06.972 WARN 3009027 --- [ocp-async-1,10fddfe6c0e76764,a335e72d8abf1b4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:06.972 INFO 3009027 --- [ocp-async-9,1583b63d00e8ba06,96a413f3c153c52c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:07.165 INFO 3009027 --- [ocp-schedules-7,4453fbab74ebb1c2,9b8688cace060027] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:37:07.167 INFO 3009027 --- [ocp-schedules-7,4453fbab74ebb1c2,9b8688cace060027] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:37:08.182 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:08.183 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:08.972 WARN 3009027 --- [ocp-async-7,9a2e912de95a1924,dde0b3864981d214] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:08.972 INFO 3009027 --- [ocp-async-10,47c45bcb490464b6,dd0fdc285b7fac01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,3f6a4ea3564d364a,0e7318911ed71104] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:10.001 INFO 3009027 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:37:10.002 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:37:10.972 WARN 3009027 --- [ocp-async-4,a530039962eb942f,fe84e0d6dde73a18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:10.972 INFO 3009027 --- [ocp-async-10,9c2cda8e16becf5a,9defe4392179576d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:10.991 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:37:11.183 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:11.183 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,0dfc5435fc18ba59,d4744f84ff89e586] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:12.408 INFO 3009027 --- [prometheus-cache-monitor1,740ab56943a8deef,d8b35bdbced7fb78] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1670,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:37:12.408 INFO 3009027 --- [prometheus-cache-monitor1,740ab56943a8deef,d8b35bdbced7fb78] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:37:12.927 INFO 3009027 --- [ocp-async-10,069d1c19e8937996,1fc55849930e7818] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:37:12.954 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:33:12.907, end=2025-04-08T20:35:12.952} 2025-04-08 20:37:12.957 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:37:12.957 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:37:12.972 WARN 3009027 --- [ocp-async-1,07eb1f759bdc263d,61487b9a4aedf8b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:12.972 INFO 3009027 --- [ocp-async-2,5ae731bd9f5ae606,a1c9bc3a85338737] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:12.975 INFO 3009027 --- [ocp-schedules-8,15c2231d1cc9716d,f02bfc140a094453] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:37:13.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:37:13.177566+08:00, lastProcessTime=2025-04-08T20:37:03.177350+08:00 2025-04-08 20:37:13.322 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:37:13.177566+08:00 2025-04-08 20:37:14.183 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:14.184 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:14.973 WARN 3009027 --- [ocp-async-5,393ab7e3c8c34d45,943b4cf9d3f841d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:14.973 INFO 3009027 --- [ocp-async-2,3a3fa617aa714869,37c492ddd3bc742c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,cc48b5a90710737c,fb6fb62841ba5f47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:15.548 INFO 3009027 --- [ocp-schedules-21,4cac027948d1b67c,953f9b927c600dfd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:37:15.582 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115760, nodeCount=67, affectRows=67 2025-04-08 20:37:15.582 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115760, nodeCount=67, validCount=67 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115760, nodeCount=81, affectRows=81 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115760, nodeCount=66, affectRows=66 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115760, nodeCount=68, affectRows=68 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115760, nodeCount=63, affectRows=63 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115760, nodeCount=73, affectRows=73 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115760, nodeCount=66, validCount=66 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115760, nodeCount=68, validCount=68 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115760, nodeCount=62, validCount=62 2025-04-08 20:37:15.589 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115760, nodeCount=79, validCount=79 2025-04-08 20:37:15.590 INFO 3009027 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115760, nodeCount=73, validCount=73 2025-04-08 20:37:16.973 WARN 3009027 --- [ocp-async-2,d5aa4ab6f03ddb6f,5d8597e4fb1325ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:16.973 INFO 3009027 --- [ocp-async-6,e9064d9351d587dd,0e71d1564fe43cfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:17.174 INFO 3009027 --- [ocp-schedules-7,090430a65817b832,da0d021068f16ecb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:37:17.175 INFO 3009027 --- [ocp-schedules-7,090430a65817b832,da0d021068f16ecb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:37:17.184 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:17.184 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,965111aeab9f8229,200c319c9c6b2d62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:18.973 WARN 3009027 --- [ocp-async-1,15589184d162ec16,7066b5953a2c9e64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:18.973 INFO 3009027 --- [ocp-async-10,24ade04aa28484f2,e0afe5c6aa3096ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:20.185 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:20.186 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:20.973 WARN 3009027 --- [ocp-async-10,ed41d938a1456b7d,4a1ca21675770383] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:20.973 INFO 3009027 --- [ocp-async-4,048269ba4531fd35,cc494aa993364cbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,3150d9e103ebe6e6,ebe0d99a1c5544ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:22.895 INFO 3009027 --- [ocp-schedules-22,9d4719bbc83d8de5,706d77922f2b7cdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:37:22.896 INFO 3009027 --- [ocp-schedules-3,20849c604669b05f,c3e9a5610fb83ac8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:37:22.896 INFO 3009027 --- [ocp-schedules-4,419d2f80613ec34d,bc77eb68a8a73e4d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:37:22.896 INFO 3009027 --- [ocp-schedules-21,4ae0c99c054cd0ce,0a5385452e9fa8f2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:37:22.896 INFO 3009027 --- [ocp-schedules-8,c19790d6b48fbc5a,23e59d2815f41b61] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:37:22.897 INFO 3009027 --- [ocp-schedules-4,419d2f80613ec34d,bc77eb68a8a73e4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:22.897 INFO 3009027 --- [ocp-schedules-8,c19790d6b48fbc5a,23e59d2815f41b61] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:37:22.898 INFO 3009027 --- [ocp-schedules-13,42249ea38b933267,77cfc4928ba22f2f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:37:22.898 INFO 3009027 --- [ocp-schedules-26,e486852af0f1fca8,5b387fee205a4c4b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:37:22.898 INFO 3009027 --- [ocp-schedules-26,e486852af0f1fca8,5b387fee205a4c4b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:37:22.898 INFO 3009027 --- [ocp-schedules-13,42249ea38b933267,77cfc4928ba22f2f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:37:22.899 INFO 3009027 --- [ocp-schedules-22,9d4719bbc83d8de5,706d77922f2b7cdd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:37:22.899 INFO 3009027 --- [ocp-schedules-24,740603b12391540f,01c7ed8e2b707131] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:37:22.900 INFO 3009027 --- [ocp-schedules-24,740603b12391540f,01c7ed8e2b707131] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:37:22.901 INFO 3009027 --- [ocp-schedules-29,ea75dc2e0850d3e0,fb03cf476193eac5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:37:22.903 INFO 3009027 --- [ocp-schedules-21,4ae0c99c054cd0ce,0a5385452e9fa8f2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:37:22.917 INFO 3009027 --- [ocp-schedules-7,48b8c8ba917041cc,59d42959ab885c54] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:37:22.918 INFO 3009027 --- [ocp-schedules-7,48b8c8ba917041cc,59d42959ab885c54] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:37:22.927 INFO 3009027 --- [ocp-async-3,1f37f925214b056b,bf0dac3bc1e30136] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:37:22.939 INFO 3009027 --- [ocp-schedules-6,bacaa8c314de635a,acbd0f7dbd1e2f82] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:37:22.939 INFO 3009027 --- [ocp-schedules-6,bacaa8c314de635a,acbd0f7dbd1e2f82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:22.940 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:37:22.949 INFO 3009027 --- [ocp-schedules-10,677776d6d3a3d649,2d5d0cb7f074247f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:37:22.949+08:00, clusters=[1] 2025-04-08 20:37:22.950 INFO 3009027 --- [ocp-schedules-10,677776d6d3a3d649,2d5d0cb7f074247f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:37:22.958 INFO 3009027 --- [ocp-schedules-27,a4207c5c73af849a,9a80bd1d3831e672] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:37:22.965 INFO 3009027 --- [ocp-schedules-2,95ea6f41d35ea4db,c37bfef673722e8b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:37:22.965+08:00, clusters=[1] 2025-04-08 20:37:22.968 INFO 3009027 --- [ocp-schedules-2,95ea6f41d35ea4db,c37bfef673722e8b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:37:22.973 WARN 3009027 --- [ocp-async-1,b308b18dade178d7,10af3d51a2ae000c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:22.973 INFO 3009027 --- [ocp-async-1,6d4c837c4bf35e89,a029d359c1a4f18d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:22.974 INFO 3009027 --- [ocp-schedules-1,163549e93b701703,53d5fca5709ff6fc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:37:22.978 INFO 3009027 --- [ocp-schedules-24,2de1793860c6a933,504072d3447eec8e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:37:22.985 INFO 3009027 --- [ocp-schedules-29,0a32e742f96c13bb,b5933cccdbded5a0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:37:22.985 INFO 3009027 --- [ocp-schedules-29,0a32e742f96c13bb,b5933cccdbded5a0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:37:22.993 INFO 3009027 --- [ocp-schedules-23,7e59bdc991669556,f02ba830dd98fac3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:37:22.993 INFO 3009027 --- [ocp-schedules-23,7e59bdc991669556,f02ba830dd98fac3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:37:22.997 INFO 3009027 --- [ocp-schedules-23,7e59bdc991669556,f02ba830dd98fac3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:37:22.997 INFO 3009027 --- [ocp-schedules-23,7e59bdc991669556,f02ba830dd98fac3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:37:23.003 INFO 3009027 --- [ocp-schedules-18,e7f6a6c517aa0244,29002ca966bb3afc] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:37:23.004 INFO 3009027 --- [ocp-schedules-18,e7f6a6c517aa0244,29002ca966bb3afc] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:37:23.006 INFO 3009027 --- [ocp-schedules-25,7b2be3606008cbd7,b75d24423d620743] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:37:23.008 INFO 3009027 --- [ocp-schedules-25,7b2be3606008cbd7,b75d24423d620743] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:37:23.008 INFO 3009027 --- [ocp-schedules-25,7b2be3606008cbd7,b75d24423d620743] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:37:23.014 INFO 3009027 --- [ocp-schedules-15,cb9e0da34cdceaf5,f5fc9a07f689c93f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:37:23.017 INFO 3009027 --- [ocp-schedules-16,1572921f19d62f04,4aedd02a05ea0369] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:37:23.017 INFO 3009027 --- [ocp-schedules-16,635c9d08c0f241d4,663533efdc0e2b1f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:37:23.017 INFO 3009027 --- [ocp-schedules-16,635c9d08c0f241d4,663533efdc0e2b1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:23.023 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:37:23.023 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:37:23.053 INFO 3009027 --- [ocp-schedules-7,7a4e75f87a8bf9ba,4cf70da4fb07aa47] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:37:23.054 INFO 3009027 --- [ocp-schedules-7,7a4e75f87a8bf9ba,4cf70da4fb07aa47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:23.070 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:37:23.071 ERROR 3009027 --- [ocp-schedules-7,7a4e75f87a8bf9ba,4cf70da4fb07aa47] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:37:23.071 INFO 3009027 --- [ocp-schedules-7,7a4e75f87a8bf9ba,4cf70da4fb07aa47] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:37:23.080 INFO 3009027 --- [ocp-schedules-9,98d8a65f63a3d763,d8abb2c0543c90c5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:37:23.097 INFO 3009027 --- [ocp-schedules-11,dd6140d1524f9d11,78a69609b1d95802] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:37:23.097 INFO 3009027 --- [ocp-schedules-11,dd6140d1524f9d11,78a69609b1d95802] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:37:23.100 INFO 3009027 --- [ocp-schedules-11,dd6140d1524f9d11,78a69609b1d95802] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:37:23.100 INFO 3009027 --- [ocp-schedules-11,dd6140d1524f9d11,78a69609b1d95802] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766199115910868992 2025-04-08 20:37:23.103 INFO 3009027 --- [ocp-schedules-11,dd6140d1524f9d11,78a69609b1d95802] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:37:23.103 INFO 3009027 --- [ocp-schedules-11,dd6140d1524f9d11,78a69609b1d95802] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:37:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:37:23.176690+08:00, lastProcessTime=2025-04-08T20:37:13.177566+08:00 2025-04-08 20:37:23.186 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:23.186 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:23.273 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:37:23.176690+08:00 2025-04-08 20:37:23.831 INFO 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:37:23.831 INFO 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:37:23.834 INFO 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:37:23.835 WARN 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:37:23.835 INFO 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:37:23.836 WARN 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:37:23.839 INFO 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:37:23.841 WARN 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:37:23.841 ERROR 3009027 --- [ocp-schedules-21,c9f5c5c890fead97,bd00c5eee5259df1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:37:23.940 INFO 3009027 --- [ocp-schedules-14,ad41439c7a793f7b,0cda3d1ef3f401ff] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:37:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,515388621b81d6d9,a44e562ec3008440] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:24.916 INFO 3009027 --- [ocp-schedules-10,0cdf282e4152182b,a9275ae63df71cbf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:37:24.950 INFO 3009027 --- [ocp-schedules-10,0cdf282e4152182b,a9275ae63df71cbf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:37:24.950 INFO 3009027 --- [ocp-schedules-10,0cdf282e4152182b,a9275ae63df71cbf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:37:24.973 WARN 3009027 --- [ocp-async-10,f3e1032e1ef18761,b5c944393020b109] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:24.973 INFO 3009027 --- [ocp-async-7,f7eab859f60edd13,9e0b766cac8dee43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:25.569 INFO 3009027 --- [ocp-schedules-9,78be856e9e831125,99a893bea250a5d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:37:26.186 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:26.187 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:26.973 WARN 3009027 --- [ocp-async-2,4cd3756af5b8cfb3,1336adf7b5e0d608] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:26.973 INFO 3009027 --- [ocp-async-4,0ce305d85d3e10e2,2c153c624e44762e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:37:27.180 INFO 3009027 --- [ocp-schedules-30,014a8538f3bc59bc,3de93da95f5aef77] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:37:27.181 INFO 3009027 --- [ocp-schedules-30,014a8538f3bc59bc,3de93da95f5aef77] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:37:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,aeb69bcc8e714544,2638a38073575979] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:28.974 WARN 3009027 --- [ocp-async-4,d3f1f2e467315d74,a1ff3fc87a2b9249] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:28.974 INFO 3009027 --- [ocp-async-8,37a49e63a16bc7bd,d1f0f09f6f74b4e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:29.187 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:29.188 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,cb4ad5cbff866d20,606f0a93c52594ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:30.974 WARN 3009027 --- [ocp-async-9,81ab5a801c9f018d,5909d61866a28072] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:30.974 INFO 3009027 --- [ocp-async-10,81e37d97a4ca003f,9e3abe4b6474844c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:32.188 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:32.188 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:32.897 INFO 3009027 --- [ocp-schedules-4,419d2f80613ec34d,bc77eb68a8a73e4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:37:32.898 INFO 3009027 --- [ocp-schedules-8,c19790d6b48fbc5a,23e59d2815f41b61] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:37:32.928 INFO 3009027 --- [ocp-async-2,63c0e1d41c0d2d39,2265f77a30e84e7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:37:32.940 INFO 3009027 --- [ocp-schedules-6,bacaa8c314de635a,acbd0f7dbd1e2f82] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:37:32.974 INFO 3009027 --- [ocp-async-5,71d6e0e17da44e38,d7cd406446aff973] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:32.974 WARN 3009027 --- [ocp-async-3,2b723f5e30e5d506,f607388829aac39d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:32.979 INFO 3009027 --- [ocp-schedules-5,0d04afab39f85e84,07dcab25031fa1aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:37:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:37:33.176870+08:00, lastProcessTime=2025-04-08T20:37:23.176690+08:00 2025-04-08 20:37:33.280 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:37:33.176870+08:00 2025-04-08 20:37:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,e752bdd8c41b4e43,d3d99a79ec6f4dfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:34.974 INFO 3009027 --- [ocp-async-6,e44112f36f7c88e0,f4c2d59056992d4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:34.974 WARN 3009027 --- [ocp-async-8,d886afefb928456e,bb5554411cafd95e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:35.189 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:35.189 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:35.594 INFO 3009027 --- [ocp-schedules-11,e3800d144cda25a3,a888de01e017f6f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:37:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,31f1b68099ee5736,04167882a7d48779] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:36.974 WARN 3009027 --- [ocp-async-1,950989b2e68a69f8,45fe1be94370201f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:36.974 INFO 3009027 --- [ocp-async-7,ece42b075232b915,af06858536caf800] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:37.188 INFO 3009027 --- [ocp-schedules-28,c0ef001d6be75598,97a0e9ddfe463d7a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:37:37.189 INFO 3009027 --- [ocp-schedules-28,c0ef001d6be75598,97a0e9ddfe463d7a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:37:38.189 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:38.190 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:38.974 WARN 3009027 --- [ocp-async-3,f7b1bd2a8625103a,feafb7c9ef850f73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:38.974 INFO 3009027 --- [ocp-async-7,40029606d88a1f00,341d37d175d2fdf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,d8d936fe451f5dd0,5082bbea63b5da36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:40.001 INFO 3009027 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:37:40.001 INFO 3009027 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:37:40.974 WARN 3009027 --- [ocp-async-5,f6048987f8b733c9,51651c297ee8f9b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:40.974 INFO 3009027 --- [ocp-async-8,24b0458fc48b67a7,2ea3479fbe8fa166] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:41.190 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:41.191 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,47b053fb3d75896a,f742e89bcec20432] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:42.928 INFO 3009027 --- [ocp-async-9,a7c35efab4b5822e,8ba99a4142de3c77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:37:42.975 WARN 3009027 --- [ocp-async-2,270a160e7ce752a8,84a4bf477c4b54a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:42.975 INFO 3009027 --- [ocp-async-10,a853d21c7370f1b5,c3e04d93d1445f0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:42.980 INFO 3009027 --- [ocp-schedules-22,91344136ac5fa669,4dbe0cc6a90999dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:37:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:37:43.176734+08:00, lastProcessTime=2025-04-08T20:37:33.176870+08:00 2025-04-08 20:37:43.260 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:37:43.176734+08:00 2025-04-08 20:37:44.191 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:44.192 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:44.952 INFO 3009027 --- [ocp-schedules-20,c14de9e575702ef2,e22e8b09b318ffa2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:37:44.975 WARN 3009027 --- [ocp-async-3,f199ce04fcab8c07,34faa2a25cae28f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:44.975 INFO 3009027 --- [ocp-async-7,c8803a3d6a495368,6e3d5a94e7282368] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:44.999 INFO 3009027 --- [ocp-schedules-20,c14de9e575702ef2,e22e8b09b318ffa2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:37:44.999 INFO 3009027 --- [ocp-schedules-20,c14de9e575702ef2,e22e8b09b318ffa2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:37:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,74bec52469533476,76458738592df61c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:45.615 INFO 3009027 --- [ocp-schedules-18,081d5e727238064c,84fc794455f5d11a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:37:46.975 WARN 3009027 --- [ocp-async-8,49e0b103dfcf490e,a6325875fbb08c42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:46.975 INFO 3009027 --- [ocp-async-6,a5eb7618d48e03e3,9a3b7dcb8dc68f3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:47.192 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:47.192 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:47.193 INFO 3009027 --- [ocp-schedules-11,08a5cf8d272aa4aa,798cc9652fdee43d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:37:47.194 INFO 3009027 --- [ocp-schedules-11,08a5cf8d272aa4aa,798cc9652fdee43d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:37:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,f33eefcfecf37a9c,2116b4ca5a8a580b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:48.975 WARN 3009027 --- [ocp-async-1,93ccd9d96901877e,22d7450d6be424dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:48.975 INFO 3009027 --- [ocp-async-2,a4e469d44e2ddf9b,ee4eb208b9142370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:50.192 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:50.193 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:50.975 WARN 3009027 --- [ocp-async-4,112c5aeca6a5d18d,c8a0f295213cbc32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:50.975 INFO 3009027 --- [ocp-async-10,e8e6a1ea33c7aa66,aa7e1f8537d38354] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,78234d4039f3f4ec,c52c28b24d096450] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:52.896 INFO 3009027 --- [ocp-schedules-19,3c9d66d407466ff7,94524110e64d1b17] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:37:52.896 INFO 3009027 --- [ocp-schedules-18,31ab9e38b963bcbc,aac3b408a7316632] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:37:52.896 INFO 3009027 --- [ocp-schedules-30,89e8852eb8c43cc9,c7423a40610ac143] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:37:52.897 INFO 3009027 --- [ocp-schedules-18,31ab9e38b963bcbc,aac3b408a7316632] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:52.897 INFO 3009027 --- [ocp-schedules-30,89e8852eb8c43cc9,c7423a40610ac143] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:52.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:37:52.899 INFO 3009027 --- [ocp-schedules-23,297f7f20b953b1c7,89e8f50568a1dbe9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:37:52.899 INFO 3009027 --- [ocp-schedules-23,297f7f20b953b1c7,89e8f50568a1dbe9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:37:52.899 INFO 3009027 --- [ocp-schedules-10,3de2dbabf00cc686,ff4e2b36997ef9f1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:37:52.899 INFO 3009027 --- [ocp-schedules-10,3de2dbabf00cc686,ff4e2b36997ef9f1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:37:52.899 INFO 3009027 --- [ocp-schedules-14,9e5b806bac534e07,c15f2e4d3507b35b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:37:52.899 INFO 3009027 --- [ocp-schedules-14,9e5b806bac534e07,c15f2e4d3507b35b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:37:52.900 INFO 3009027 --- [ocp-schedules-27,02efa76341f8ea89,81385c7a098fd53e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:37:52.928 INFO 3009027 --- [ocp-async-10,46caaf6422d8bbdc,b056477885df3062] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:37:52.975 INFO 3009027 --- [ocp-schedules-15,cf834fd79e98cc7f,8f9780ad4cb5ae36] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:37:52.975 WARN 3009027 --- [ocp-async-5,91be0627154b4c06,64e341e482989e0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:52.975 INFO 3009027 --- [ocp-async-8,29b30d38dbb21227,590baf03ede5356b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:52.981 INFO 3009027 --- [ocp-schedules-20,2bbbc23dd3f06f01,d994fe90aae3ef83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:37:53.023 INFO 3009027 --- [ocp-schedules-16,635c9d08c0f241d4,663533efdc0e2b1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:37:53.023 INFO 3009027 --- [ocp-schedules-16,8708cbf12d683d97,043a1510695b8c87] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:37:53.024 INFO 3009027 --- [ocp-schedules-16,8708cbf12d683d97,043a1510695b8c87] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:37:53.029 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:37:53.029 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:37:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:37:53.177289+08:00, lastProcessTime=2025-04-08T20:37:43.176734+08:00 2025-04-08 20:37:53.193 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:53.194 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:53.305 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:37:53.177289+08:00 2025-04-08 20:37:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,c2532dc199961bb2,4fcccea0e881e980] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:54.975 WARN 3009027 --- [ocp-async-9,43efa376d5d973e8,2fdd411d0a410b28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:54.976 INFO 3009027 --- [ocp-async-8,70b4c8fca43d724e,a90f66110786a412] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:55.642 INFO 3009027 --- [ocp-schedules-25,116bc3392987732a,a8f73d17c1fbe713] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:37:56.194 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:56.194 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:37:56.976 WARN 3009027 --- [ocp-async-4,6df3ac6ab54bf801,ea3cc40b59ef0e2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:56.976 INFO 3009027 --- [ocp-async-7,652b092bc3712539,e93c23e00fc80caf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:57.197 INFO 3009027 --- [ocp-schedules-7,99447d119fe10602,f36c97ff016d68f4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:37:57.198 INFO 3009027 --- [ocp-schedules-7,99447d119fe10602,f36c97ff016d68f4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:37:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,8395d244209fa1c2,9d30ac8139ef3b62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115880, interval=60 2025-04-08 20:37:58.976 WARN 3009027 --- [ocp-async-5,d6b23990cd1a2a26,ea0241a0ab9a8397] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:37:58.976 INFO 3009027 --- [ocp-async-6,ecc024d9698dbebf,9244284d225ccfd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:37:59.194 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:37:59.195 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,3651f26bd153c935,a8e5d752a52e5f46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:00.976 WARN 3009027 --- [ocp-async-9,789affbea508fd79,2a4554c12dfb2e54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:00.976 INFO 3009027 --- [ocp-async-10,4148cd6d9b9e11cb,dafe58af842ae1bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:02.195 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:02.196 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:02.897 INFO 3009027 --- [ocp-schedules-18,31ab9e38b963bcbc,aac3b408a7316632] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:38:02.898 INFO 3009027 --- [ocp-schedules-30,89e8852eb8c43cc9,c7423a40610ac143] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:38:02.928 INFO 3009027 --- [ocp-async-8,5c269158a06fb543,e2619167e2b22d6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:38:02.976 WARN 3009027 --- [ocp-async-4,84dcfa1a995f07eb,801a89cfc9907b59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:02.976 INFO 3009027 --- [ocp-async-7,aec33a54c3951515,7ea52d6682a09697] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:02.982 INFO 3009027 --- [ocp-schedules-22,3cd5e42c4572b443,230788e28c169837] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:38:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:38:03.177501+08:00, lastProcessTime=2025-04-08T20:37:53.177289+08:00 2025-04-08 20:38:03.353 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:38:03.177501+08:00 2025-04-08 20:38:03.410 INFO 3009027 --- [prometheus-minute-schedule-1,237b8a9639cd11fe,b48c14585d1883ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:04.976 WARN 3009027 --- [ocp-async-1,77bb7cba7c961e51,eba460c13ad08572] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:04.976 INFO 3009027 --- [ocp-async-6,70af7a188b1d4ff9,9481e132c762808d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:05.001 INFO 3009027 --- [ocp-schedules-1,918c730ccb2cc216,dd3e1a3a7d8604cd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:38:05.010 INFO 3009027 --- [ocp-schedules-1,918c730ccb2cc216,dd3e1a3a7d8604cd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:38:05.010 INFO 3009027 --- [ocp-schedules-1,918c730ccb2cc216,dd3e1a3a7d8604cd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:38:05.196 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:05.197 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:05.661 INFO 3009027 --- [ocp-schedules-12,622ec0cd5175adbf,b42608f48efb217f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:38:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,9efdded0db508145,891c946d35574c8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:06.976 WARN 3009027 --- [ocp-async-6,653fb943b782c5d2,a6360edec67a724f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:06.976 INFO 3009027 --- [ocp-async-9,c7f0795809f69b3d,1343e4f80cb0daf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:07.300 INFO 3009027 --- [ocp-schedules-3,930eb5112c71b823,cf54d3aa499570f9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:38:07.301 INFO 3009027 --- [ocp-schedules-3,930eb5112c71b823,cf54d3aa499570f9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:38:08.197 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:08.198 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:08.977 WARN 3009027 --- [ocp-async-4,dcf13655a643e5a6,ccab2e2b2f2ca79d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:08.977 INFO 3009027 --- [ocp-async-10,009a0f461411a815,b78d28d79661d091] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,d7dc166f6d1feef8,d609338977afa502] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:10.002 INFO 3009027 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:38:10.002 INFO 3009027 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:38:10.977 WARN 3009027 --- [ocp-async-10,31ede43e55faec02,30043fd4db5e0c04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:10.977 INFO 3009027 --- [ocp-async-1,24de1aaef5c0ce7e,67274adfeadabe9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:10.988 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:38:11.198 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:11.199 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,0d6cce2ddc86b919,d641f54717e0ac29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:12.408 INFO 3009027 --- [prometheus-cache-monitor1,72df56a5d9278099,16c337b3c9e77178] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1672,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:38:12.408 INFO 3009027 --- [prometheus-cache-monitor1,72df56a5d9278099,16c337b3c9e77178] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:38:12.928 INFO 3009027 --- [ocp-async-1,d39358a959a5c6f4,8eb0069fbb7e974e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:38:12.977 WARN 3009027 --- [ocp-async-6,6292c34f894bd30f,3e412290c6fabc82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:12.977 INFO 3009027 --- [ocp-async-2,24b4a5753b6ed70f,ec6f4e9d33e4ed51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:12.984 INFO 3009027 --- [ocp-schedules-2,63b0c48d5c2e9001,cf5f36ebb122bf57] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:38:13.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:38:13.177378+08:00, lastProcessTime=2025-04-08T20:38:03.177501+08:00 2025-04-08 20:38:13.319 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:38:13.177378+08:00 2025-04-08 20:38:14.199 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:14.200 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:14.977 WARN 3009027 --- [ocp-async-3,4c087401267c9a2c,32fc65e8d9e10d96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:14.977 INFO 3009027 --- [ocp-async-9,f3bac4157994f148,c95422d307f43e94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,9fa861a73a9e223f,49a0ae1d6ee5c132] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:15.597 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115820, nodeCount=418, affectRows=418 2025-04-08 20:38:15.600 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115820, nodeCount=415, validCount=415 2025-04-08 20:38:15.686 INFO 3009027 --- [ocp-schedules-14,a109dc196d2a65b9,ad6a77670fadbb04] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:38:16.977 WARN 3009027 --- [ocp-async-9,7456f48a0a372c62,2f6e4a1d67ef8c38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:16.977 INFO 3009027 --- [ocp-async-5,6aeacb82ec226aa6,0a685308bc97ce59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:17.200 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:17.200 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:17.314 INFO 3009027 --- [ocp-schedules-11,1e28178a2d508391,1865bdd2d8989b03] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:38:17.316 INFO 3009027 --- [ocp-schedules-11,1e28178a2d508391,1865bdd2d8989b03] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:38:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,2affced42c35149f,17e6cb399acfd945] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:18.977 WARN 3009027 --- [ocp-async-4,1b11eb0c2a1af656,98b22103cf2a7c13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:18.977 INFO 3009027 --- [ocp-async-6,0966f3f620313dbf,336ea13accd1f591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:20.200 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:20.201 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:20.977 WARN 3009027 --- [ocp-async-3,b53a022a24e557c6,04f5ebdef301ff40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:20.977 INFO 3009027 --- [ocp-async-10,e515e1663b907a13,698c132eff1adf8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,07c20395214cb0d9,8c879f00c1d13159] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:22.895 INFO 3009027 --- [ocp-schedules-17,9034915053725f29,2addca8eaa02b4a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:38:22.896 INFO 3009027 --- [ocp-schedules-2,f574b4225025bb68,26fbb89287bd20ac] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:38:22.896 INFO 3009027 --- [ocp-schedules-10,ded42a946d04d68f,1a96fcff5b67663c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:38:22.897 INFO 3009027 --- [ocp-schedules-6,f9af02eacfaf3d45,1bd01f269e94ab59] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:38:22.897 INFO 3009027 --- [ocp-schedules-7,d3811660de8ed780,dd79134d3282627a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:38:22.897 INFO 3009027 --- [ocp-schedules-10,ded42a946d04d68f,1a96fcff5b67663c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:22.897 INFO 3009027 --- [ocp-schedules-6,f9af02eacfaf3d45,1bd01f269e94ab59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:38:22.898 INFO 3009027 --- [ocp-schedules-9,6a4d7366f7388fc9,a041d01a5fe15b3c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:38:22.898 INFO 3009027 --- [ocp-schedules-9,6a4d7366f7388fc9,a041d01a5fe15b3c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:38:22.898 INFO 3009027 --- [ocp-schedules-22,01b2fc0b39093fff,c86c00688019358e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:38:22.898 INFO 3009027 --- [ocp-schedules-22,01b2fc0b39093fff,c86c00688019358e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:38:22.899 INFO 3009027 --- [ocp-schedules-13,48fab6d6aebdd01f,2f71bf521576432b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:38:22.899 INFO 3009027 --- [ocp-schedules-13,48fab6d6aebdd01f,2f71bf521576432b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:38:22.904 INFO 3009027 --- [ocp-schedules-7,d3811660de8ed780,dd79134d3282627a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:38:22.904 INFO 3009027 --- [ocp-schedules-17,9034915053725f29,2addca8eaa02b4a3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:38:22.907 INFO 3009027 --- [ocp-schedules-27,0f11f60a61dbf750,5c879a09f9a929ad] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:38:22.914 INFO 3009027 --- [ocp-schedules-15,9e4f18408a095d52,d77b5f49fe0b15b0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1672,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":416,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:38:22.914 INFO 3009027 --- [ocp-schedules-15,9e4f18408a095d52,d77b5f49fe0b15b0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:38:22.928 INFO 3009027 --- [ocp-async-1,985eb3929cfa98ee,166de038f21680ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:38:22.945 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:38:22.945 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:38:22.950 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:38:22.950 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:38:22.950 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:38:22.952 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:38:22.952 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:38:22.952 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:38:22.954 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:38:22.954 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:38:22.954 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:38:22.955 INFO 3009027 --- [ocp-schedules-25,c07710dbab132321,8a224e5f214a30c0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:38:22.955+08:00, clusters=[1] 2025-04-08 20:38:22.955 INFO 3009027 --- [ocp-schedules-25,c07710dbab132321,8a224e5f214a30c0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:38:22.956 INFO 3009027 --- [ocp-schedules-14,1dce32624741765d,dc368bd1e59bd208] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:38:22.959 INFO 3009027 --- [ocp-schedules-21,0f897aa2deaa5ced,5decb5ed5565b2aa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:38:22.973 INFO 3009027 --- [ocp-schedules-30,9e3f60ff88ccc41c,d876e8ef1c953b41] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:38:22.973+08:00, clusters=[1] 2025-04-08 20:38:22.974 INFO 3009027 --- [ocp-schedules-30,9e3f60ff88ccc41c,d876e8ef1c953b41] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:38:22.976 INFO 3009027 --- [ocp-schedules-28,a7654b501294d8fa,a63654d62c42113e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:38:22.977 WARN 3009027 --- [ocp-async-7,4c0eed676357424a,0b00e81480402e67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:22.978 INFO 3009027 --- [ocp-async-7,d8008e2fa3b96740,b6d6d96521289648] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:22.985 INFO 3009027 --- [ocp-schedules-22,ca5b9e4ffdff8198,0336869e3ace78d3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:38:22.985 INFO 3009027 --- [ocp-schedules-13,bacdd1be5c02f5e7,e0a64a23105a72f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:38:22.985 INFO 3009027 --- [ocp-schedules-22,ca5b9e4ffdff8198,0336869e3ace78d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:22.987 INFO 3009027 --- [ocp-schedules-7,ca7fb9c12bca50a8,b73c7cab2567213f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:38:22.987 INFO 3009027 --- [ocp-schedules-7,ca7fb9c12bca50a8,b73c7cab2567213f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:38:22.996 ERROR 3009027 --- [ocp-schedules-22,ca5b9e4ffdff8198,0336869e3ace78d3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:38:22.996 INFO 3009027 --- [ocp-schedules-22,ca5b9e4ffdff8198,0336869e3ace78d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:38:22.997 INFO 3009027 --- [ocp-schedules-4,58e29dcdd993a166,9ef5e15b5e18cd15] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:38:22.997 INFO 3009027 --- [ocp-schedules-4,58e29dcdd993a166,9ef5e15b5e18cd15] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:38:23.002 INFO 3009027 --- [ocp-schedules-4,58e29dcdd993a166,9ef5e15b5e18cd15] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:38:23.002 INFO 3009027 --- [ocp-schedules-4,58e29dcdd993a166,9ef5e15b5e18cd15] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:38:23.005 INFO 3009027 --- [ocp-schedules-15,2f2c3498cf6ba7ad,2184784b5d7b68fd] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:38:23.005 INFO 3009027 --- [ocp-schedules-15,2f2c3498cf6ba7ad,2184784b5d7b68fd] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:38:23.008 INFO 3009027 --- [ocp-schedules-24,f80828e69bb7b180,14f5d80385cf5959] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:38:23.009 INFO 3009027 --- [ocp-schedules-24,f80828e69bb7b180,14f5d80385cf5959] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:38:23.009 INFO 3009027 --- [ocp-schedules-24,f80828e69bb7b180,14f5d80385cf5959] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:38:23.014 INFO 3009027 --- [ocp-schedules-29,ef8f5e283059c015,89c3d8bb486119b7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:38:23.014 INFO 3009027 --- [ocp-schedules-29,ef8f5e283059c015,89c3d8bb486119b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:23.028 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:38:23.028 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:38:23.029 INFO 3009027 --- [ocp-schedules-16,8708cbf12d683d97,043a1510695b8c87] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:38:23.030 INFO 3009027 --- [ocp-schedules-16,f14b46f4d4d818a0,001902f9394f2db3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:38:23.030 INFO 3009027 --- [ocp-schedules-16,f14b46f4d4d818a0,001902f9394f2db3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:23.035 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:38:23.035 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:38:23.071 INFO 3009027 --- [ocp-schedules-19,174852faa1a1dbcc,3f8093afd2b75a0d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:38:23.072 INFO 3009027 --- [ocp-schedules-19,174852faa1a1dbcc,3f8093afd2b75a0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:23.085 INFO 3009027 --- [ocp-schedules-11,a9e25cb67ffc710c,9d643fb16efbff37] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:38:23.088 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:38:23.089 ERROR 3009027 --- [ocp-schedules-19,174852faa1a1dbcc,3f8093afd2b75a0d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:38:23.089 INFO 3009027 --- [ocp-schedules-19,174852faa1a1dbcc,3f8093afd2b75a0d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:38:23.106 INFO 3009027 --- [ocp-schedules-1,6f2d29e521257c1d,b6bb6e5d4fe0fbfa] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:38:23.106 INFO 3009027 --- [ocp-schedules-1,6f2d29e521257c1d,b6bb6e5d4fe0fbfa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:38:23.106 INFO 3009027 --- [ocp-schedules-1,6f2d29e521257c1d,b6bb6e5d4fe0fbfa] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:38:23.106 INFO 3009027 --- [ocp-schedules-1,6f2d29e521257c1d,b6bb6e5d4fe0fbfa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766199115910868992 2025-04-08 20:38:23.107 INFO 3009027 --- [ocp-schedules-1,6f2d29e521257c1d,b6bb6e5d4fe0fbfa] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:38:23.107 INFO 3009027 --- [ocp-schedules-1,6f2d29e521257c1d,b6bb6e5d4fe0fbfa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:38:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:38:23.176333+08:00, lastProcessTime=2025-04-08T20:38:13.177378+08:00 2025-04-08 20:38:23.201 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:23.202 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:23.284 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:38:23.176333+08:00 2025-04-08 20:38:23.844 INFO 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:38:23.844 INFO 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:38:23.847 INFO 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:38:23.848 WARN 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:38:23.848 INFO 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:38:23.849 WARN 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:38:23.852 INFO 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:38:23.854 WARN 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:38:23.854 ERROR 3009027 --- [ocp-schedules-22,c44cdcbc4151fd0a,73ff546ca8776ce0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:38:24.023 INFO 3009027 --- [ocp-schedules-27,9df2d24a59c2dbb8,8e3bd70b1daa9ecf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:38:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,8c96af928167f479,5cb867d6b81c2f1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:24.978 WARN 3009027 --- [ocp-async-7,891aa7ce3b220b9b,d1f7e557518fbd92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:24.978 INFO 3009027 --- [ocp-async-3,4da77526d5d27e27,456963eeaeb2eadf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:25.011 INFO 3009027 --- [ocp-schedules-20,586b912d0c382da9,207720c8215efe32] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:38:25.015 INFO 3009027 --- [ocp-schedules-20,586b912d0c382da9,207720c8215efe32] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:38:25.015 INFO 3009027 --- [ocp-schedules-20,586b912d0c382da9,207720c8215efe32] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:38:25.705 INFO 3009027 --- [ocp-schedules-4,5c5eb2d170497b49,3d1f50fa28f32a5f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:38:26.202 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:26.202 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:38:26.978 WARN 3009027 --- [ocp-async-2,1d6d4014be07aa8f,d6f6fbbe65ad40c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:26.978 INFO 3009027 --- [ocp-async-10,4dc9940132efa027,77e1a533fa5f4d7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,49e9449180cbdf4f,7bfebbd5e6b348a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:27.428 INFO 3009027 --- [ocp-schedules-3,f65113698d623725,9d406bfc97b8d95d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:38:27.429 INFO 3009027 --- [ocp-schedules-3,f65113698d623725,9d406bfc97b8d95d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:38:28.978 WARN 3009027 --- [ocp-async-10,82a43a98576f31bf,59544434f57e06f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:28.978 INFO 3009027 --- [ocp-async-8,c0e5c59304f877a8,83943715991359a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:29.202 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:29.203 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,a8b4853c4a3a97fe,bc43658c9b0c6690] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:30.978 WARN 3009027 --- [ocp-async-7,f60e7b122f13a11e,6c56e195dc28d7ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:30.978 INFO 3009027 --- [ocp-async-6,d8811b78a492cd76,61fd8cdbf1f4d734] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:32.203 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:32.204 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:32.898 INFO 3009027 --- [ocp-schedules-10,ded42a946d04d68f,1a96fcff5b67663c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:38:32.898 INFO 3009027 --- [ocp-schedules-6,f9af02eacfaf3d45,1bd01f269e94ab59] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:38:32.929 INFO 3009027 --- [ocp-async-2,3e62955bfba44255,215d84d7664308ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:38:32.940 INFO 3009027 --- [ocp-schedules-12,d8c4e4de0f23271b,a8aa7cce7ca62e4e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:38:32.941 INFO 3009027 --- [ocp-schedules-12,d8c4e4de0f23271b,a8aa7cce7ca62e4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:32.941 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:38:32.978 INFO 3009027 --- [ocp-async-1,5ccdb2a7913e00c6,861eed1cb4e67a19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:32.978 WARN 3009027 --- [ocp-async-9,8df9cbdb0c948756,bccefbd835c11484] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:32.986 INFO 3009027 --- [ocp-schedules-28,a439ee0d4f4f0e0d,7ae8caa0db57071e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:38:33.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:38:33.175711+08:00, lastProcessTime=2025-04-08T20:38:23.176333+08:00 2025-04-08 20:38:33.286 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:38:33.175711+08:00 2025-04-08 20:38:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,1a8519bbeed5852d,663b13137c7952ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:34.978 WARN 3009027 --- [ocp-async-4,52fa39ee80ccdf72,26e13a1e99aa0a29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:34.978 INFO 3009027 --- [ocp-async-1,70291b61d13efb78,8dfc529b2c420991] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:35.204 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:35.205 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:35.738 INFO 3009027 --- [ocp-schedules-13,6b99f9ccd8dcdbef,d292d5ddf285baae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:38:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,52a097fd80d012aa,db1060d5a9a4c613] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:36.979 WARN 3009027 --- [ocp-async-3,35ff9ce8a247c109,5728fcc03bdfacab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:36.979 INFO 3009027 --- [ocp-async-6,cffb00fc5f1bd410,459fabe20ed44c53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:37.489 INFO 3009027 --- [ocp-schedules-27,c0a3358b01143d4f,d88866b553c31119] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:38:37.490 INFO 3009027 --- [ocp-schedules-27,c0a3358b01143d4f,d88866b553c31119] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:38:38.205 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:38.206 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:38.979 WARN 3009027 --- [ocp-async-10,3041ab2c08eb5c57,3658d8ca577ed156] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:38.979 INFO 3009027 --- [ocp-async-9,fc2be9e76b0c6404,b184de1f6ac1c078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,cb1c224c04cda0c6,b940f5994a9f50f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:40.001 INFO 3009027 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:38:40.001 INFO 3009027 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:38:40.953 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:38:40.958 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1), status=TAKINGOVER 2025-04-08 20:38:40.958 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 20:38:40.979 WARN 3009027 --- [ocp-async-9,ecdd505b34452a58,f27f762fa9d553f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:40.979 INFO 3009027 --- [ocp-async-4,5fd7ee798aece15c,f36aec7a6519e407] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:41.206 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:41.206 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,7a6d17b74b528f29,25ae948df6fee9fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:42.929 INFO 3009027 --- [ocp-async-1,31cc12ddfd89fda0,2cd91f8e8ad1eae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:38:42.941 INFO 3009027 --- [ocp-schedules-12,d8c4e4de0f23271b,a8aa7cce7ca62e4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:38:42.979 WARN 3009027 --- [ocp-async-3,f1d2546e2235b6d0,7470e4c3643d4132] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:42.979 INFO 3009027 --- [ocp-async-6,4cd173ea83bcacfb,a63b3cd2e32f48c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:42.987 INFO 3009027 --- [ocp-schedules-2,f7dcb5e298550be8,9eed56921ae7f8d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:38:43.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:38:43.176516+08:00, lastProcessTime=2025-04-08T20:38:33.175711+08:00 2025-04-08 20:38:43.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:38:43.176516+08:00 2025-04-08 20:38:44.207 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:44.207 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:44.979 WARN 3009027 --- [ocp-async-10,e6cef84528294908,c35ac791cdf14e2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:44.979 INFO 3009027 --- [ocp-async-5,bf0642bfc542c9a6,f3f6c0427a384b37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:45.016 INFO 3009027 --- [ocp-schedules-3,6da54a05096cd18d,09b6f6fa90a753ec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:38:45.037 INFO 3009027 --- [ocp-schedules-3,6da54a05096cd18d,09b6f6fa90a753ec] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:38:45.037 INFO 3009027 --- [ocp-schedules-3,6da54a05096cd18d,09b6f6fa90a753ec] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:38:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,522e85eab3fcd704,ec3598d236c032e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:45.759 INFO 3009027 --- [ocp-schedules-28,710f36e0807bc0d6,cd0b4c1b531b756d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:38:46.979 WARN 3009027 --- [ocp-async-4,e9a7a64a65dd1452,9bde73ede2b5b58a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:46.979 INFO 3009027 --- [ocp-async-9,9a607f614a6eb73c,4e03c261263f347e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:47.207 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:47.208 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:47.528 INFO 3009027 --- [ocp-schedules-30,b86f391c82e11412,c3aeea3e7993717e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:38:47.529 INFO 3009027 --- [ocp-schedules-30,b86f391c82e11412,c3aeea3e7993717e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:38:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,b0d361724b89dc09,21676b3a57f6a35e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:48.979 WARN 3009027 --- [ocp-async-9,8840a70b635d9a6e,68c368c1f548b75c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:48.980 INFO 3009027 --- [ocp-async-1,09350dae0e1c97ea,503f78789cdd4e32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:50.208 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:50.209 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:50.980 WARN 3009027 --- [ocp-async-10,0bc6c355848b8307,b07a5970fbb977d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:50.980 INFO 3009027 --- [ocp-async-2,7420ed9970fdac6e,bd701530eb067553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,339136ff16ef0799,f3cd905dba158325] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:52.896 INFO 3009027 --- [ocp-schedules-9,8063d0725ea9fd77,f1a9231012e0e373] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:38:52.896 INFO 3009027 --- [ocp-schedules-14,cf1dd391f9ac816e,f1f0003d807a1c99] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:38:52.896 INFO 3009027 --- [ocp-schedules-27,eba0d64039fa7db7,1e8d27f0ff40696c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:38:52.897 INFO 3009027 --- [ocp-schedules-9,8063d0725ea9fd77,f1a9231012e0e373] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:52.897 INFO 3009027 --- [ocp-schedules-27,eba0d64039fa7db7,1e8d27f0ff40696c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:38:52.898 INFO 3009027 --- [ocp-schedules-12,6a0405c30842ca2b,2965599c05f5590e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:38:52.898 INFO 3009027 --- [ocp-schedules-1,acfadf82b2b6c1d9,94449b1dc8d5c49c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:38:52.898 INFO 3009027 --- [ocp-schedules-7,3500956ec8da78a7,9b2b3b88c6734fc4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:38:52.899 INFO 3009027 --- [ocp-schedules-7,3500956ec8da78a7,9b2b3b88c6734fc4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:38:52.899 INFO 3009027 --- [ocp-schedules-1,acfadf82b2b6c1d9,94449b1dc8d5c49c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:38:52.899 INFO 3009027 --- [ocp-schedules-12,6a0405c30842ca2b,2965599c05f5590e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:38:52.900 INFO 3009027 --- [ocp-schedules-10,9820a91c42f8e977,4b3aaed6a069b80b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:38:52.929 INFO 3009027 --- [ocp-async-2,90872e4eec1d9108,39d9672fbe45443c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:38:52.978 INFO 3009027 --- [ocp-schedules-5,e9ed903887c40f4a,efc16c416f0526c8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:38:52.980 WARN 3009027 --- [ocp-async-4,d29d393ba804592b,37f24f495fd36962] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:52.980 INFO 3009027 --- [ocp-async-7,d021d3b64e8a5dc4,c420a892425e9535] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:52.989 INFO 3009027 --- [ocp-schedules-13,0f33123ffe23dc3b,1bf3764119153198] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:38:53.036 INFO 3009027 --- [ocp-schedules-16,f14b46f4d4d818a0,001902f9394f2db3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:38:53.036 INFO 3009027 --- [ocp-schedules-16,be0e190ef4f5ca1b,34b1c17a916cb6b0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:38:53.037 INFO 3009027 --- [ocp-schedules-16,be0e190ef4f5ca1b,34b1c17a916cb6b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:38:53.042 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:38:53.042 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:38:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:38:53.176832+08:00, lastProcessTime=2025-04-08T20:38:43.176516+08:00 2025-04-08 20:38:53.209 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:53.209 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:53.261 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:38:53.176832+08:00 2025-04-08 20:38:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,6fb84e0a4de6ff9c,21b1151078069ef5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:54.980 WARN 3009027 --- [ocp-async-9,24e3bfadfe9d6a30,abfc0d22388921f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:54.980 INFO 3009027 --- [ocp-async-8,249f867fc29a4fe3,c4386b4cbff34a77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:55.786 INFO 3009027 --- [ocp-schedules-23,e6d89d406bb429b7,2524864bea3d8158] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:38:56.209 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:56.210 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:38:56.980 INFO 3009027 --- [ocp-async-10,5078c2e5f5176800,998b2ab55ba2261f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:56.980 WARN 3009027 --- [ocp-async-5,586f1d73b3d354a2,3d725a5c230b14b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,9adc75b8922f4b9d,4299d9853266e4c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744115940, interval=60 2025-04-08 20:38:57.541 INFO 3009027 --- [ocp-schedules-12,9e1c3acb14c8d0af,77e2990fb3cf6276] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:38:57.543 INFO 3009027 --- [ocp-schedules-12,9e1c3acb14c8d0af,77e2990fb3cf6276] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:38:58.980 WARN 3009027 --- [ocp-async-2,986d3a30af75af4e,6b3beb11981cb80f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:38:58.980 INFO 3009027 --- [ocp-async-3,2668d9d9b9718026,ac0b5a487793642e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:38:59.210 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:38:59.211 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,2eac9a4039c8abbe,271190e51e5249a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:00.980 WARN 3009027 --- [ocp-async-7,bf470b54c8fbbacf,468064af28264eb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:00.980 INFO 3009027 --- [ocp-async-1,376f48d03f4eb15d,da793550317867bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:02.211 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:02.212 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:02.897 INFO 3009027 --- [ocp-schedules-9,8063d0725ea9fd77,f1a9231012e0e373] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:39:02.898 INFO 3009027 --- [ocp-schedules-27,eba0d64039fa7db7,1e8d27f0ff40696c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:39:02.929 INFO 3009027 --- [ocp-async-8,e4df5a10aa60ca8d,8576bf64a0eb0ddb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:39:02.980 WARN 3009027 --- [ocp-async-10,8b25ae1fd140599b,8d8765a30870c149] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:02.980 INFO 3009027 --- [ocp-async-5,684ba1b25d112247,9c273dcc6c0ad791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:02.990 INFO 3009027 --- [ocp-schedules-14,c6dc81e8125088d0,2c5aa70c832f5238] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:39:03.179 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:39:03.178810+08:00, lastProcessTime=2025-04-08T20:38:53.176832+08:00 2025-04-08 20:39:03.280 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:39:03.178810+08:00 2025-04-08 20:39:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,74f0da3bd4e3b4d7,982c7098121addc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:04.981 WARN 3009027 --- [ocp-async-4,482f003683e32b13,5f60121ef18bf7ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:04.981 INFO 3009027 --- [ocp-async-3,7783d5977361e09e,3dc2cdfbe123db9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:05.039 INFO 3009027 --- [ocp-schedules-30,e3c3447e70932855,24e46614af15303b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:39:05.043 INFO 3009027 --- [ocp-schedules-30,e3c3447e70932855,24e46614af15303b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:39:05.043 INFO 3009027 --- [ocp-schedules-30,e3c3447e70932855,24e46614af15303b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:39:05.212 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:05.213 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:05.808 INFO 3009027 --- [ocp-schedules-10,8dbd27cc310df6fc,7da003e81ce0a55e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:39:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,e10ef14098eb505d,89275b43030c403a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:06.981 WARN 3009027 --- [ocp-async-7,9c51fa46f22ea41b,ab4b86306d2d176a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:06.981 INFO 3009027 --- [ocp-async-9,b386953348e33bd6,7b053e2ff59118ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:07.657 INFO 3009027 --- [ocp-schedules-13,328b06c4727241d4,24a8215d505f9e89] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:39:07.658 INFO 3009027 --- [ocp-schedules-13,328b06c4727241d4,24a8215d505f9e89] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:39:08.213 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:08.213 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:08.981 WARN 3009027 --- [ocp-async-9,5cd36bc84b1a84ce,8812d14dc65c1b6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:08.981 INFO 3009027 --- [ocp-async-1,591691462141f8ff,e71a878ed726f60c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,1495b62072af0b61,35113f27892703d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:10.002 INFO 3009027 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:39:10.002 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:39:10.981 WARN 3009027 --- [ocp-async-5,421fd72ffa9a168e,70921b7537e6709a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:10.981 INFO 3009027 --- [ocp-async-1,8e292eeb4a27b33c,148af50fcc5394bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:10.990 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:39:11.214 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:11.214 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,62698568fca71ff2,7123a2849759d75d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:12.408 INFO 3009027 --- [prometheus-cache-monitor1,3f4e59dd461c3e0b,abd643c1132ccc07] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1672,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:39:12.408 INFO 3009027 --- [prometheus-cache-monitor1,3f4e59dd461c3e0b,abd643c1132ccc07] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:39:12.929 INFO 3009027 --- [ocp-async-7,5bf6cc0bc8ee8b3d,5baf6c2a90f7f762] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:39:12.981 WARN 3009027 --- [ocp-async-1,a66484156feff0c6,c71bbfabd8f55e7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:12.981 INFO 3009027 --- [ocp-async-8,0deeed46b392f9aa,8547a2452a2407ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:12.984 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:35:12.952, end=2025-04-08T20:37:12.976} 2025-04-08 20:39:12.986 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:39:12.986 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:39:12.991 INFO 3009027 --- [ocp-schedules-4,00e4bde12b8a598b,4c742038ea03c576] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:39:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:39:13.176830+08:00, lastProcessTime=2025-04-08T20:39:03.178810+08:00 2025-04-08 20:39:13.287 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:39:13.176830+08:00 2025-04-08 20:39:14.214 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:14.215 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:14.981 WARN 3009027 --- [ocp-async-6,e935772c720bc8ef,933c720221e7b36a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:14.981 INFO 3009027 --- [ocp-async-10,f294737aebf06331,76aa7d2bded0dcef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,b46073f1b52591b5,b4adb30cafce8ff4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:15.598 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115880, nodeCount=76, affectRows=76 2025-04-08 20:39:15.599 INFO 3009027 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115880, nodeCount=75, validCount=75 2025-04-08 20:39:15.614 INFO 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115880, nodeCount=87, affectRows=87 2025-04-08 20:39:15.614 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115880, nodeCount=80, affectRows=80 2025-04-08 20:39:15.614 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115880, nodeCount=95, affectRows=95 2025-04-08 20:39:15.614 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115880, nodeCount=80, affectRows=80 2025-04-08 20:39:15.615 INFO 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115880, nodeCount=87, validCount=87 2025-04-08 20:39:15.615 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115880, nodeCount=79, validCount=79 2025-04-08 20:39:15.615 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115880, nodeCount=79, validCount=79 2025-04-08 20:39:15.615 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115880, nodeCount=95, validCount=95 2025-04-08 20:39:15.839 INFO 3009027 --- [ocp-schedules-25,e4ea94cd199e3592,738a998e11912ae2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:39:16.982 WARN 3009027 --- [ocp-async-10,25314b50041c3d13,9762e5d04f236a7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:16.982 INFO 3009027 --- [ocp-async-2,0768b44da235c94f,a8959350f194aa8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:17.215 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:17.216 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:17.662 INFO 3009027 --- [ocp-schedules-8,06176369c6a99973,9b6caebedde55d7b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:39:17.663 INFO 3009027 --- [ocp-schedules-8,06176369c6a99973,9b6caebedde55d7b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:39:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,b082ced79f485cb2,b5f31fef65898b8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:18.982 WARN 3009027 --- [ocp-async-3,bc638c05cd1d69bb,a1bd464f5ac51d52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:18.982 INFO 3009027 --- [ocp-async-2,699281d16dc1deb9,0fc50934162749f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:20.216 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:20.217 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:20.982 WARN 3009027 --- [ocp-async-2,16035351c73ff096,1dacb2c54d407ad3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:20.982 INFO 3009027 --- [ocp-async-5,61567aa3f925ea69,fee653c00bd033d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,e6383270838ffdab,4b834e88371708d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:22.896 INFO 3009027 --- [ocp-schedules-8,e3663d7e007bd22e,b35ee245719cabf8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:39:22.896 INFO 3009027 --- [ocp-schedules-15,f06f0efaffe63507,6dbe408821cfc29b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:39:22.896 INFO 3009027 --- [ocp-schedules-28,d1bc7c4550a6941e,a44266d2c07861c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:39:22.897 INFO 3009027 --- [ocp-schedules-18,7bcf4d1a76aa5425,1c7f4494bd8d9b97] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:39:22.897 INFO 3009027 --- [ocp-schedules-15,f06f0efaffe63507,6dbe408821cfc29b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:22.897 INFO 3009027 --- [ocp-schedules-18,7bcf4d1a76aa5425,1c7f4494bd8d9b97] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:22.897 INFO 3009027 --- [ocp-schedules-23,684c6131943f8775,eab6a42854a4df25] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:39:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:39:22.898 INFO 3009027 --- [ocp-schedules-30,5e5001545853df04,737735216e2c790f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:39:22.898 INFO 3009027 --- [ocp-schedules-30,5e5001545853df04,737735216e2c790f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:39:22.899 INFO 3009027 --- [ocp-schedules-3,2646f53007230017,7b9814b9012ef8d1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:39:22.899 INFO 3009027 --- [ocp-schedules-12,6de419e6e9a448e8,af5d1e161a5fa862] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:39:22.899 INFO 3009027 --- [ocp-schedules-3,2646f53007230017,7b9814b9012ef8d1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:39:22.899 INFO 3009027 --- [ocp-schedules-12,6de419e6e9a448e8,af5d1e161a5fa862] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:39:22.900 INFO 3009027 --- [ocp-schedules-10,c64fd91270fb2b56,727b7986a6c0cc67] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:39:22.903 INFO 3009027 --- [ocp-schedules-5,35615617ddd85f1d,b183a63988964176] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 20:39:22.903 INFO 3009027 --- [ocp-schedules-5,35615617ddd85f1d,b183a63988964176] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 20:39:22.904 INFO 3009027 --- [ocp-schedules-23,684c6131943f8775,eab6a42854a4df25] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:39:22.914 INFO 3009027 --- [pool-overview-container-6,491a159241e2b4f7,521d53f2bd0f2bd8] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:39:22.914 INFO 3009027 --- [pool-overview-container-8,df52164eb191404f,561934afdfcdd638] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:39:22.919 INFO 3009027 --- [ocp-schedules-2,3d8dfb6dcf5b0a6f,72bad52388394c06] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:39:22.920 INFO 3009027 --- [pool-11-thread-14,df52164eb191404f,f3ec5d605b66463e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 20:39:22.920 INFO 3009027 --- [pool-11-thread-13,df52164eb191404f,91fa6647563cfb03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 20:39:22.920 INFO 3009027 --- [pool-11-thread-13,df52164eb191404f,91fa6647563cfb03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 20:39:22.920 INFO 3009027 --- [ocp-schedules-2,3d8dfb6dcf5b0a6f,72bad52388394c06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:22.920 INFO 3009027 --- [pool-11-thread-15,df52164eb191404f,7d98af09618a60ad] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 20:39:22.921 INFO 3009027 --- [ocp-schedules-28,d1bc7c4550a6941e,a44266d2c07861c3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:39:22.922 INFO 3009027 --- [ForkJoinPool-7-worker-2,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:39:22.922 INFO 3009027 --- [pool-11-thread-16,df52164eb191404f,d6e692dbbc11034d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 20:39:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-2,df52164eb191404f,28edbc5fe4b32c86] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 20:39:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-2,df52164eb191404f,28edbc5fe4b32c86] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 20:39:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-2,df52164eb191404f,28edbc5fe4b32c86] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 20:39:22.923 INFO 3009027 --- [pool-11-thread-15,df52164eb191404f,7d98af09618a60ad] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 20:39:22.924 INFO 3009027 --- [pool-11-thread-13,df52164eb191404f,91fa6647563cfb03] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 20:39:22.924 INFO 3009027 --- [pool-11-thread-13,df52164eb191404f,91fa6647563cfb03] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 20:39:22.925 INFO 3009027 --- [ForkJoinPool-7-worker-11,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:39:22.943 INFO 3009027 --- [pool-11-thread-14,df52164eb191404f,f3ec5d605b66463e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 20:39:22.943 INFO 3009027 --- [pool-11-thread-14,df52164eb191404f,f3ec5d605b66463e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 20:39:22.943 INFO 3009027 --- [ocp-async-7,12bb54753b481e83,661c0ecb2cb647f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:39:22.943 INFO 3009027 --- [ocp-schedules-6,bfc74bc44eaa3c36,da305c2c3bb3c169] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:39:22.943 INFO 3009027 --- [ForkJoinPool-7-worker-11,df52164eb191404f,9c0b4e5514fb36e5] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 20:39:22.943 INFO 3009027 --- [pool-11-thread-16,df52164eb191404f,d6e692dbbc11034d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 20:39:22.949 INFO 3009027 --- [ocp-schedules-6,bfc74bc44eaa3c36,da305c2c3bb3c169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:39:22.950 WARN 3009027 --- [ocp-schedules-6,bfc74bc44eaa3c36,da305c2c3bb3c169] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:39:22.951 INFO 3009027 --- [ocp-schedules-6,bfc74bc44eaa3c36,da305c2c3bb3c169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:39:22.952 WARN 3009027 --- [ocp-schedules-6,bfc74bc44eaa3c36,da305c2c3bb3c169] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:39:22.955 INFO 3009027 --- [ocp-schedules-6,bfc74bc44eaa3c36,da305c2c3bb3c169] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=1 2025-04-08 20:39:22.960 INFO 3009027 --- [ocp-schedules-21,7fd9f37bf0f23088,602f76e1c078652a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:39:22.960+08:00, clusters=[1] 2025-04-08 20:39:22.960 INFO 3009027 --- [ocp-schedules-21,7fd9f37bf0f23088,602f76e1c078652a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:39:22.960 INFO 3009027 --- [ocp-schedules-30,691ae63913b0614c,f5f97182275759ef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:39:22.973 INFO 3009027 --- [ocp-schedules-3,449001aa5f24db60,57995a080026f4e0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:39:22.973 INFO 3009027 --- [ocp-schedules-3,449001aa5f24db60,57995a080026f4e0] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:39:22.979 INFO 3009027 --- [pool-23-thread-1,449001aa5f24db60,b2604334bbd9d336] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 20:39:22.980 INFO 3009027 --- [ocp-schedules-10,90a8e4fd7cf4298e,d08056c712eff9a7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:39:22.980 INFO 3009027 --- [ocp-schedules-12,51dd6a8679f16177,0687f7f3b0d7f8ae] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:39:22.980+08:00, clusters=[1] 2025-04-08 20:39:22.982 INFO 3009027 --- [ocp-schedules-12,51dd6a8679f16177,0687f7f3b0d7f8ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:39:22.982 WARN 3009027 --- [ocp-async-4,050d90e7f90707c2,b59457d64b8ee802] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:22.982 INFO 3009027 --- [ocp-async-9,e3f0625da3498ce1,0a93ae804123f9ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:22.985 WARN 3009027 --- [ocp-schedules-3,449001aa5f24db60,57995a080026f4e0] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$ece00efc.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:39:22.985 INFO 3009027 --- [ocp-schedules-3,449001aa5f24db60,57995a080026f4e0] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:39:22.990 INFO 3009027 --- [ocp-schedules-5,864ce5beb6bc795d,cb821c6517fabbb1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:39:22.990 INFO 3009027 --- [ocp-schedules-5,864ce5beb6bc795d,cb821c6517fabbb1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:39:22.992 INFO 3009027 --- [ocp-schedules-26,692c59804074e50c,b76ff8a6090e74b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:39:23.003 INFO 3009027 --- [ocp-schedules-24,8a515ca306354b9a,1821664215bfb1da] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:39:23.003 INFO 3009027 --- [ocp-schedules-24,8a515ca306354b9a,1821664215bfb1da] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:39:23.008 INFO 3009027 --- [ocp-schedules-9,023b87c02a2a0abb,ebfb84e0d2e0994d] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:39:23.008 INFO 3009027 --- [ocp-schedules-9,023b87c02a2a0abb,ebfb84e0d2e0994d] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:39:23.009 INFO 3009027 --- [ocp-schedules-24,8a515ca306354b9a,1821664215bfb1da] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:39:23.009 INFO 3009027 --- [ocp-schedules-24,8a515ca306354b9a,1821664215bfb1da] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:39:23.010 INFO 3009027 --- [ocp-schedules-7,578d6422ebb46a47,e7ff37451da5c23e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:39:23.011 INFO 3009027 --- [ocp-schedules-7,578d6422ebb46a47,e7ff37451da5c23e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:39:23.011 INFO 3009027 --- [ocp-schedules-7,578d6422ebb46a47,e7ff37451da5c23e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:39:23.033 INFO 3009027 --- [ocp-schedules-29,ef8f5e283059c015,89c3d8bb486119b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:39:23.043 INFO 3009027 --- [ocp-schedules-16,be0e190ef4f5ca1b,34b1c17a916cb6b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:39:23.043 INFO 3009027 --- [ocp-schedules-16,f5732716ca03f0fe,62bcfa56eac343b7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:39:23.043 INFO 3009027 --- [ocp-schedules-16,f5732716ca03f0fe,62bcfa56eac343b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:23.049 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:39:23.049 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:39:23.090 INFO 3009027 --- [ocp-schedules-17,bd25bddff6f1910c,854396788a27871e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:39:23.090 INFO 3009027 --- [ocp-schedules-17,bd25bddff6f1910c,854396788a27871e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:23.092 INFO 3009027 --- [ocp-schedules-22,1d3399ab04ac9909,9c37c6cd4e0f0194] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:39:23.106 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:39:23.107 ERROR 3009027 --- [ocp-schedules-17,bd25bddff6f1910c,854396788a27871e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:39:23.107 INFO 3009027 --- [ocp-schedules-17,bd25bddff6f1910c,854396788a27871e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:39:23.108 INFO 3009027 --- [ocp-schedules-8,0d1017729d0983ee,324c19deac1fdb76] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:39:23.108 INFO 3009027 --- [ocp-schedules-8,0d1017729d0983ee,324c19deac1fdb76] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:39:23.108 INFO 3009027 --- [ocp-schedules-8,0d1017729d0983ee,324c19deac1fdb76] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:39:23.108 INFO 3009027 --- [ocp-schedules-8,0d1017729d0983ee,324c19deac1fdb76] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766199115910868992 2025-04-08 20:39:23.108 INFO 3009027 --- [ocp-schedules-8,0d1017729d0983ee,324c19deac1fdb76] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:39:23.108 INFO 3009027 --- [ocp-schedules-8,0d1017729d0983ee,324c19deac1fdb76] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:39:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:39:23.176713+08:00, lastProcessTime=2025-04-08T20:39:13.176830+08:00 2025-04-08 20:39:23.217 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:23.217 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:23.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:39:23.176713+08:00 2025-04-08 20:39:23.857 INFO 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:39:23.857 INFO 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-04-08 20:39:23.860 INFO 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:39:23.861 WARN 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:39:23.861 INFO 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:39:23.862 WARN 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:39:23.865 INFO 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:39:23.867 WARN 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:39:23.867 ERROR 3009027 --- [ocp-schedules-5,7577cf6bd68aef72,9bb431397f21272a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 20:39:24.039 INFO 3009027 --- [ocp-schedules-13,a6c33951adcaec3c,3cf36cf0efe845ab] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:39:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,32a6920a8df66dac,29a1435008123623] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:24.982 WARN 3009027 --- [ocp-async-9,0c75c13a9c61675c,882981292c7f88d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:24.982 INFO 3009027 --- [ocp-async-2,f6e9b78872953e44,25825d66b377e4d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:25.044 INFO 3009027 --- [ocp-schedules-23,2ef30c817f4697b1,c6414e4d97efe4f8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:39:25.047 INFO 3009027 --- [ocp-schedules-23,2ef30c817f4697b1,c6414e4d97efe4f8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:39:25.047 INFO 3009027 --- [ocp-schedules-23,2ef30c817f4697b1,c6414e4d97efe4f8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:39:25.861 INFO 3009027 --- [ocp-schedules-10,80780d8806ec62d9,b57fc7f12dad5928] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:39:26.217 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:26.218 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:39:26.982 WARN 3009027 --- [ocp-async-10,129c0530bb8e370a,34d7970ccb197159] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:26.982 INFO 3009027 --- [ocp-async-6,dca1ea2f597d0d6a,9dab4529cd923c52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,27aa401631a47c3e,65312ba2d93d2ad3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:27.710 INFO 3009027 --- [ocp-schedules-5,dc54a36ce0bf05a1,10708aeeb8645afd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:39:27.712 INFO 3009027 --- [ocp-schedules-5,dc54a36ce0bf05a1,10708aeeb8645afd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:39:28.982 WARN 3009027 --- [ocp-async-6,e675eb782678e59b,fa1cdd60215d8b69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:28.982 INFO 3009027 --- [ocp-async-4,cf947fdbf19d235e,13d6bf763b31907b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:29.218 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:29.219 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:29.348 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:39:29.348 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:39:29.352 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:39:29.352 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:39:29.352 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:39:29.352 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:39:29.355 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:39:29.355 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:39:29.355 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:39:29.355 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:39:29.378 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:39:29.382 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:39:29.382 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:39:29.382 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:39:29.382 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:39:29.382 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:39:29.388 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:39:29.687 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:39:30.027 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.67181593S 2025-04-08 20:39:30.027 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:39:30.087 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:39:30.087 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:39:30.111 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:39:30.149 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:39:30.149 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:39:30.149 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:39:30.149 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:39:30.149 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:39:30.149 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:39:30.156 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:39:30.253 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] o.s.boot.SpringApplication : Started application in 0.901 seconds (JVM running for 1011.87) 2025-04-08 20:39:30.253 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.901494996S 2025-04-08 20:39:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,e14ee53d17efd730,63806ca6200e35ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:30.509 INFO 3009027 --- [ocp-schedules-10,d7fe5140cc31f527,65a5270b9532f014] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:39:30.983 WARN 3009027 --- [ocp-async-3,39420858e0ab6a13,fc4207a7dcf6959e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:30.983 INFO 3009027 --- [ocp-async-8,1bc11e8bf7055854,6f0fb80bfcf87fd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:32.219 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:32.219 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:32.897 INFO 3009027 --- [ocp-schedules-15,f06f0efaffe63507,6dbe408821cfc29b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:39:32.897 INFO 3009027 --- [ocp-schedules-18,7bcf4d1a76aa5425,1c7f4494bd8d9b97] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:39:32.920 INFO 3009027 --- [ocp-schedules-2,3d8dfb6dcf5b0a6f,72bad52388394c06] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 20:39:32.943 INFO 3009027 --- [ocp-async-10,0211bbd3b1b0be89,465add287daaca96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:39:32.983 WARN 3009027 --- [ocp-async-2,da574ce5a9ca06ab,f9cfaeb21cf9b9a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:32.983 INFO 3009027 --- [ocp-async-6,89ed5b0d6388942b,475233cb96a5ea2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:32.993 INFO 3009027 --- [ocp-schedules-1,a62ddd38fd6130d9,73223230eb249437] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:39:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:39:33.175971+08:00, lastProcessTime=2025-04-08T20:39:23.176713+08:00 2025-04-08 20:39:33.261 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:39:33.175971+08:00 2025-04-08 20:39:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,068334b07de14a87,1870efffd3dd9f5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:34.983 WARN 3009027 --- [ocp-async-6,2183b26de58e0294,b32bb676f2cc2318] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:34.983 INFO 3009027 --- [ocp-async-3,b16488fef1fd4ef9,236cc0c2524fce4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:35.220 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:35.220 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:35.888 INFO 3009027 --- [ocp-schedules-3,29d97d16f18c5705,67220a338b5d4a86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:39:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,fe2ef133e9fbf014,45c59251846ce98c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:36.983 WARN 3009027 --- [ocp-async-3,f609a7971d9febe7,38f3bba62db7ed1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:36.983 INFO 3009027 --- [ocp-async-5,09c2ebd141468650,d1171275f78d82e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:37.728 INFO 3009027 --- [ocp-schedules-27,a298b70e3a799f25,96fde733ed697d56] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:39:37.729 INFO 3009027 --- [ocp-schedules-27,a298b70e3a799f25,96fde733ed697d56] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:39:38.220 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:38.221 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:38.983 WARN 3009027 --- [ocp-async-1,9b1c27baa09978d1,336f73d492fd9d66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:38.983 INFO 3009027 --- [ocp-async-7,09cc867c336328ab,2ef0be49b97a2972] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,bad4cfa1d9c7eae1,cdfea72af4ea25b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:40.001 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:39:40.001 INFO 3009027 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:39:40.983 WARN 3009027 --- [ocp-async-6,63469cbae62311c4,918dd66cf2861fa0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:40.983 INFO 3009027 --- [ocp-async-7,840c96a55065b2df,b6701acc0d1db061] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:41.221 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:41.222 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,9ae4438bf3ef425a,cd9a87a7ff91a44b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:42.941 INFO 3009027 --- [ocp-schedules-8,473a63949e4f9cf5,2b7591593146623f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:39:42.942 INFO 3009027 --- [ocp-schedules-8,473a63949e4f9cf5,2b7591593146623f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:42.943 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:39:42.943 INFO 3009027 --- [ocp-async-9,b2163f8f5a8ac798,b6556f7e7874f724] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:39:42.984 WARN 3009027 --- [ocp-async-8,7f959a6835bacf07,644dccee733dc8ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:42.984 INFO 3009027 --- [ocp-async-7,8dbc670f8b86f23b,0114a96298f812c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:42.995 INFO 3009027 --- [ocp-schedules-2,e4d8a125e530c874,235cb9532fb02601] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:39:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:39:43.177546+08:00, lastProcessTime=2025-04-08T20:39:33.175971+08:00 2025-04-08 20:39:43.303 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:39:43.177546+08:00 2025-04-08 20:39:44.222 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:44.223 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:44.984 WARN 3009027 --- [ocp-async-10,19c9dc5a700b90cf,9615d992876abe70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:44.984 INFO 3009027 --- [ocp-async-7,f6b2cff5f202564d,1466e82e680c0287] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:45.048 INFO 3009027 --- [ocp-schedules-28,19fe765c2f178e41,1b1d6ea5801f5ba0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:39:45.051 INFO 3009027 --- [ocp-schedules-28,19fe765c2f178e41,1b1d6ea5801f5ba0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:39:45.051 INFO 3009027 --- [ocp-schedules-28,19fe765c2f178e41,1b1d6ea5801f5ba0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:39:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,e0456d248c3144d1,40ecbb364d59790f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:45.908 INFO 3009027 --- [ocp-schedules-15,c65855caa5e9cbd5,a71c6b739c3b36e7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:39:46.984 WARN 3009027 --- [ocp-async-7,fe4262ab28e200ec,c2f1a263d6f97643] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:46.984 INFO 3009027 --- [ocp-async-4,ccc21c63ef566914,02cd40e89d2fac61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:47.223 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:47.223 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:47.784 INFO 3009027 --- [ocp-schedules-23,6a9756d7878f8e4b,a33142102bdc187f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:39:47.785 INFO 3009027 --- [ocp-schedules-23,6a9756d7878f8e4b,a33142102bdc187f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:39:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,73feb4eabdfa0349,02d3af7582cdbba8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:48.984 WARN 3009027 --- [ocp-async-9,bdb3e301e8cd563a,9e733caa8bf9ae9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:48.984 INFO 3009027 --- [ocp-async-4,156adaddfb309ea7,a1fb21fbb1add9d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:50.223 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:50.224 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:50.984 WARN 3009027 --- [ocp-async-5,cd6ca9eb591ad6fe,9c5446b2c92e98d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:50.984 INFO 3009027 --- [ocp-async-10,0878e1b83574bfdf,dd5cfd781881f204] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,a7d984324995f7cf,4103f5550e1e74e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:52.896 INFO 3009027 --- [ocp-schedules-28,f47fb679fd64264e,683ab942b4b29d82] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:39:52.896 INFO 3009027 --- [ocp-schedules-17,2121f9dee9144996,1e31b80c41b8964f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:39:52.896 INFO 3009027 --- [ocp-schedules-17,2121f9dee9144996,1e31b80c41b8964f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:52.896 INFO 3009027 --- [ocp-schedules-20,3037671e57589ef9,25748eedce1aeaf0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:39:52.897 INFO 3009027 --- [ocp-schedules-20,3037671e57589ef9,25748eedce1aeaf0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:39:52.898 INFO 3009027 --- [ocp-schedules-11,eb9f616f48febfd0,0b79650f668eb314] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:39:52.898 INFO 3009027 --- [ocp-schedules-11,eb9f616f48febfd0,0b79650f668eb314] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:39:52.898 INFO 3009027 --- [ocp-schedules-4,af9768ca5bd7b016,2a1129248cef0f68] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:39:52.898 INFO 3009027 --- [ocp-schedules-4,af9768ca5bd7b016,2a1129248cef0f68] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:39:52.898 INFO 3009027 --- [ocp-schedules-29,183bbd9a6f3ebc44,815b3105ef50f420] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:39:52.898 INFO 3009027 --- [ocp-schedules-29,183bbd9a6f3ebc44,815b3105ef50f420] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:39:52.900 INFO 3009027 --- [ocp-schedules-6,2f7f463483460620,e88a379ebce53870] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:39:52.943 INFO 3009027 --- [ocp-schedules-8,473a63949e4f9cf5,2b7591593146623f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:39:52.943 INFO 3009027 --- [ocp-async-10,148d877276273b09,03e1ea56643e2bd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:39:52.981 INFO 3009027 --- [ocp-schedules-25,1d13a6b5d95ecab6,a0bd6860bed9599f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:39:52.984 WARN 3009027 --- [ocp-async-8,5d2d3c02ed3491af,97c85adc4041ae77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:52.984 INFO 3009027 --- [ocp-async-7,96d023f3fd6aaea7,8a7df46404afb49f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:52.996 INFO 3009027 --- [ocp-schedules-22,c1c601ff0d82989f,89f53aba604a3ccb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:39:53.049 INFO 3009027 --- [ocp-schedules-16,f5732716ca03f0fe,62bcfa56eac343b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:39:53.049 INFO 3009027 --- [ocp-schedules-16,d38aae8ca49e0530,f289699c0a1adb6d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:39:53.050 INFO 3009027 --- [ocp-schedules-16,d38aae8ca49e0530,f289699c0a1adb6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:39:53.056 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:39:53.056 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:39:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:39:53.177773+08:00, lastProcessTime=2025-04-08T20:39:43.177546+08:00 2025-04-08 20:39:53.224 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:53.225 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:53.293 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:39:53.177773+08:00 2025-04-08 20:39:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,da062cc559b2ac0a,bc2459d1abd55406] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:54.984 WARN 3009027 --- [ocp-async-4,0ff841715bc3f351,068d8df10030a352] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:54.984 INFO 3009027 --- [ocp-async-9,0ef265c910ca0a24,0b417b0a9d91037a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:55.938 INFO 3009027 --- [ocp-schedules-23,c79f8ac65a4e5df0,5a8bdc1ab2de8ba8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:39:56.225 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:56.226 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:39:56.985 WARN 3009027 --- [ocp-async-5,cf07c7c470fd0f21,f77d78afc81e32ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:56.985 INFO 3009027 --- [ocp-async-10,6fc0ffb039ef4b0b,d44149801248866a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,3872a9f51e99d97a,1ee67b2fbb856dae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116000, interval=60 2025-04-08 20:39:57.797 INFO 3009027 --- [ocp-schedules-6,30e68481bd173077,c20b81ed4a4c2644] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:39:57.798 INFO 3009027 --- [ocp-schedules-6,30e68481bd173077,c20b81ed4a4c2644] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:39:58.985 WARN 3009027 --- [ocp-async-8,f08c1c6079da1e00,4f4e14d925828087] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:39:58.985 INFO 3009027 --- [ocp-async-3,e420931904f06260,4e92962b542eb27c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:39:59.226 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:39:59.226 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:00.209 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:35:00.0, nextRun=2025-04-08 20:40:00.0, currentDate=Tue Apr 08 20:40:00 CST 2025 2025-04-08 20:40:00.209 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:35:00.0, nextRunTime=2025-04-08 20:40:00.0} 2025-04-08 20:40:00.211 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 20:40:00.212 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:40:00.212+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:40:00.252 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:35:00.0, nextRun=2025-04-08 20:40:00.0, currentDate=Tue Apr 08 20:40:00 CST 2025 2025-04-08 20:40:00.252 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:35:00.0, nextRunTime=2025-04-08 20:40:00.0} 2025-04-08 20:40:00.257 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 20:40:00.275 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 20:40:00.275 INFO 3009027 --- [task-coordinator1,1308668168866431,4b61f41888444b48] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 20:40:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,a1722ac6c8c1811a,3c0cf68c4edad648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:00.421 INFO 3009027 --- [subtask-coordinator3,1308668168866431,928d9f15cffabd82] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 44, operation:EXECUTE, state: RUNNING 2025-04-08 20:40:00.584 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:40:00.585 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=44, context=Context{parallelIdx=-1, stringMap={task_instance_id=6, task_operation=execute, latest_execution_start_time=2025-04-08T20:40:00.416+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=44}, listMap={}}, executor=192.168.1.61 2025-04-08 20:40:00.590 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.592 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 20:40:00.592 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.592 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.592 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.593 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.593 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.594 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.594 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:40:00.598 INFO 3009027 --- [subtask-executor5,1308668168866431,c2c8d0756cf58b57] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 44, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:40:00.985 INFO 3009027 --- [ocp-async-4,f90347b9f545b442,ec4444e6c0a375bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:00.985 WARN 3009027 --- [ocp-async-1,5d667079fc506046,9b5e312eab351257] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:01.450 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:6, status:SUCCESS 2025-04-08 20:40:01.451 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:6, status:SUCCESS 2025-04-08 20:40:02.227 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:02.227 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:02.897 INFO 3009027 --- [ocp-schedules-17,2121f9dee9144996,1e31b80c41b8964f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:40:02.897 INFO 3009027 --- [ocp-schedules-20,3037671e57589ef9,25748eedce1aeaf0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:40:02.943 INFO 3009027 --- [ocp-async-4,830cd89380b675a8,571e6b55d81d4a7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:40:02.985 WARN 3009027 --- [ocp-async-2,de97e1f38fb7a4f5,9d3eea38d7485220] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:02.985 INFO 3009027 --- [ocp-async-5,497727706530ed33,75b6aed613e75741] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:02.997 INFO 3009027 --- [ocp-schedules-6,e47e49e587584b06,ec2ae29cbb3949e5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:40:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:40:03.177221+08:00, lastProcessTime=2025-04-08T20:39:53.177773+08:00 2025-04-08 20:40:03.341 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:40:03.177221+08:00 2025-04-08 20:40:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,65ff9938f7952309,6f36f2372bf0b9b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:04.709 INFO 3009027 --- [task-coordinator1,1308668168866431,f5ff74351573fdf5] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:30:05.0, nextRun=2025-04-08 20:40:04.0, currentDate=Tue Apr 08 20:40:04 CST 2025 2025-04-08 20:40:04.709 INFO 3009027 --- [task-coordinator1,1308668168866431,f5ff74351573fdf5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-04-08 20:30:05.0, nextRunTime=2025-04-08 20:40:04.0} 2025-04-08 20:40:04.712 INFO 3009027 --- [task-coordinator1,1308668168866431,f5ff74351573fdf5] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 20:40:04.712 INFO 3009027 --- [task-coordinator1,1308668168866431,f5ff74351573fdf5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:40:04.712+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:40:04.985 WARN 3009027 --- [ocp-async-3,08f54a344c8b7b4b,2207f787a0a6edff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:04.985 INFO 3009027 --- [ocp-async-8,c2321c73009f5073,cf844d4d5f9c71a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:05.052 INFO 3009027 --- [ocp-schedules-12,205dc447d016d224,089fbe93f0c4f8ff] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:40:05.058 INFO 3009027 --- [ocp-schedules-12,205dc447d016d224,089fbe93f0c4f8ff] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:40:05.058 INFO 3009027 --- [ocp-schedules-12,205dc447d016d224,089fbe93f0c4f8ff] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:40:05.227 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:05.228 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:05.588 INFO 3009027 --- [subtask-coordinator3,1308668168866431,e2faf1a6728e71bc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 45, operation:EXECUTE, state: RUNNING 2025-04-08 20:40:05.614 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:40:05.616 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=45, context=Context{parallelIdx=-1, stringMap={task_instance_id=7, task_operation=execute, latest_execution_start_time=2025-04-08T20:40:05.587+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=45}, listMap={}}, executor=192.168.1.61 2025-04-08 20:40:05.621 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 20:40:05.622 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 20:40:05.622 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.623 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:40:05.623 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.624 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.624 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.624 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 20:40:05.628 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.628 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.629 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.629 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 20:40:05.631 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 20:40:05.632 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 20:40:05.632 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.632 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:40:05.633 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.633 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.634 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.634 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 20:40:05.645 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.646 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.646 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.647 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 20:40:05.657 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.658 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.658 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.658 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 20:40:05.669 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.670 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.670 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.670 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 20:40:05.681 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.681 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.682 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.682 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 20:40:05.687 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.688 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.688 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.688 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 20:40:05.692 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.692 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.692 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.692 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 20:40:05.696 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.697 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.697 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.697 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 20:40:05.702 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.703 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.703 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.703 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 20:40:05.708 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.708 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.708 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.709 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 20:40:05.713 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.713 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.714 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.715 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 20:40:05.728 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.728 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.729 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.729 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 20:40:05.741 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.741 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.742 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.742 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 20:40:05.746 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.747 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.747 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.748 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 20:40:05.755 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.755 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.755 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.756 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 20:40:05.760 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.760 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.761 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.761 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 20:40:05.764 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.765 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.765 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.765 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 20:40:05.769 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.769 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.769 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.769 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 20:40:05.773 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.773 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.773 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.774 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 20:40:05.777 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.778 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.778 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.778 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 20:40:05.781 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.781 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.782 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.782 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 20:40:05.786 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.787 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.787 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.787 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 20:40:05.790 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.790 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.791 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.791 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 20:40:05.794 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.794 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.795 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.795 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 20:40:05.799 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.799 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.799 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.799 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 20:40:05.803 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.803 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.803 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.804 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 20:40:05.807 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.807 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.808 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.808 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 20:40:05.811 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.811 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.812 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.812 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 20:40:05.815 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.815 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.816 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.816 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 20:40:05.819 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.820 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.820 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.820 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 20:40:05.823 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.824 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.824 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.824 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 20:40:05.828 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.828 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.828 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.828 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 20:40:05.831 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.832 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.832 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.832 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 20:40:05.836 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.836 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.836 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.836 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 20:40:05.842 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.842 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.842 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.842 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 20:40:05.847 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.847 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.848 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.848 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 20:40:05.852 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.852 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.853 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.853 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 20:40:05.857 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.858 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.858 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.859 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 20:40:05.862 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.862 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.862 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.863 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 20:40:05.866 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.866 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.867 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.867 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 20:40:05.870 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.870 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.870 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.870 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 20:40:05.873 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.873 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.873 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.874 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 20:40:05.881 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.882 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.882 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.882 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 20:40:05.901 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.901 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.902 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.902 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 20:40:05.908 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.908 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.908 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.908 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 20:40:05.913 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.914 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.914 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.914 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 20:40:05.919 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.919 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.920 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.920 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 20:40:05.923 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.923 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.924 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.924 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 20:40:05.927 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.927 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:40:05.927 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:05.928 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 20:40:05.931 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 20:40:05.959 INFO 3009027 --- [ocp-schedules-20,d2f345a7b9c17c10,b7f3ebba882693d0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:40:06.055 INFO 3009027 --- [subtask-executor6,1308668168866431,b83ac1b9817dac19] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 45, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:40:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,a3abce15db1455af,5d87494c0b0580b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:06.582 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:7, status:SUCCESS 2025-04-08 20:40:06.583 INFO 3009027 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:7, status:SUCCESS 2025-04-08 20:40:06.985 WARN 3009027 --- [ocp-async-1,da9bb36c6da22a94,715f52eb1e2cb69b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:06.985 INFO 3009027 --- [ocp-async-7,b53da1dde14d0b9e,1cd7d0cc65f15199] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:07.887 INFO 3009027 --- [ocp-schedules-11,f8c63a2b86cfc788,e26a7c045510006e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:40:07.888 INFO 3009027 --- [ocp-schedules-11,f8c63a2b86cfc788,e26a7c045510006e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:40:08.228 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:08.229 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:08.985 WARN 3009027 --- [ocp-async-7,684018cd62d81ac1,d8000ae1d6bea569] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:08.985 INFO 3009027 --- [ocp-async-9,14239002209f2cb3,a83b0ee0542ad205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,d9f17333b1095508,5d2688966e5cbe9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:09.937 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,f707d63af7b9de39,0a7e1e3a60af52de] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/2/rollback, client 10.11.12.102, traceId f707d63af7b9de39] 2025-04-08 20:40:09.940 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,f707d63af7b9de39,0a7e1e3a60af52de] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/2/rollback?taskInstanceId=2, client=10.11.12.102, traceId=f707d63af7b9de39, method=NoDataResponse com.oceanbase.ocp.server.common.controller.TaskController.rollbackTask(Long), args=2,] 2025-04-08 20:40:09.947 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,f707d63af7b9de39,0a7e1e3a60af52de] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 26, current state: FAILED, new state: READY 2025-04-08 20:40:09.962 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,f707d63af7b9de39,0a7e1e3a60af52de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/2/rollback client=10.11.12.102, traceId=f707d63af7b9de39, duration=27 ms] 2025-04-08 20:40:10.002 INFO 3009027 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:40:10.002 INFO 3009027 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:40:10.214 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,a45481aff0e469ba,2bfea7bc91f83a60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=a45481aff0e469ba, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:10.223 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,a45481aff0e469ba,2bfea7bc91f83a60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=a45481aff0e469ba, duration=13 ms] 2025-04-08 20:40:10.611 INFO 3009027 --- [subtask-coordinator3,1308668168866431,200dd867f7db2f4e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=26, original=READY 2025-04-08 20:40:10.613 INFO 3009027 --- [subtask-coordinator3,1308668168866431,200dd867f7db2f4e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:10.635 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,f738bcc0843cf338,2e6a0836a1600da0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=f738bcc0843cf338, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:40:10.638 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,f738bcc0843cf338,2e6a0836a1600da0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=f738bcc0843cf338, duration=7 ms] 2025-04-08 20:40:10.709 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:10.714 WARN 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=26, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:10.611+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=26}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:10.725 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 20:40:10.749 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:40:10.754 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:40:10.764 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:40:10.787 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:10.829 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:40:10.874 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:10.907 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:40:10.933 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:10.936 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:10.974 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:40:10.985 INFO 3009027 --- [ocp-async-9,c3668feed0d4d0cd,7af67799672d17dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:10.985 WARN 3009027 --- [ocp-async-3,b0a1b61f83b788a2,3e5bb4173858dee9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:10.991 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:40:10.993 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:10.998 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:40:11.026 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.098 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.104 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:40:11.123 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.198 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:40:11.229 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:11.230 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:11.223 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.250 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.254 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 20:40:11.264 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:40:11.282 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.288 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:40:11.557 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1ee3821664457e8d,06c708bfb5a240c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=1ee3821664457e8d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:11.567 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,1ee3821664457e8d,06c708bfb5a240c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=1ee3821664457e8d, duration=14 ms] 2025-04-08 20:40:11.658 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.660 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.662 WARN 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:40:11.664 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:40:11.665 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 20:40:11.698 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:40:11.774 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ce33d09794f2707,724436afadeffc11] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=6ce33d09794f2707, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:40:11.721 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:11.802 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ce33d09794f2707,724436afadeffc11] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=6ce33d09794f2707, duration=32 ms] 2025-04-08 20:40:11.801 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:40:12.169 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:12.186 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:40:12.188 WARN 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 20:40:12.190 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 20:40:12.191 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 20:40:12.199 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 26, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:12.209 INFO 3009027 --- [manual-subtask-executor15,73ac1c39155401d0,9e12139c3b3d0e8e] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[19] 2025-04-08 20:40:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,19f4ceff3bd30b32,77b4337727564296] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:12.408 INFO 3009027 --- [prometheus-cache-monitor1,885d765015b4697f,a3497145cfae0489] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1672,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":419,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:40:12.408 INFO 3009027 --- [prometheus-cache-monitor1,885d765015b4697f,a3497145cfae0489] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=62, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=62, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:40:12.462 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T20:34:12.36, end=2025-04-08T20:39:12.46} 2025-04-08 20:40:12.463 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T20:34:12.403, end=2025-04-08T20:39:12.461} 2025-04-08 20:40:12.467 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 20:40:12.467 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:40:12.467 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 20:40:12.467 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:40:12.703 INFO 3009027 --- [subtask-coordinator3,1308668168866431,25c2181a3051cd97] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=19, original=READY 2025-04-08 20:40:12.704 INFO 3009027 --- [subtask-coordinator3,1308668168866431,25c2181a3051cd97] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 19, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:12.792 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,7732f0165e4f41f2,2be6043feaaa7ca9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=7732f0165e4f41f2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:12.802 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,7732f0165e4f41f2,2be6043feaaa7ca9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=7732f0165e4f41f2, duration=14 ms] 2025-04-08 20:40:12.844 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,6536d1825e034ba7] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:12.852 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,6536d1825e034ba7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=19, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:12.703+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=19}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:12.860 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,6536d1825e034ba7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 19, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:12.862 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,6536d1825e034ba7] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[3] 2025-04-08 20:40:12.944 INFO 3009027 --- [ocp-async-9,ed2717894ddacc56,0a17a2841c43bc12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:40:12.986 INFO 3009027 --- [ocp-async-1,da9e4b16891d5001,bf0ddac9dfcf4fa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:12.986 WARN 3009027 --- [ocp-async-4,86416c1c00d625b2,fe976c7652adf27d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:12.998 INFO 3009027 --- [ocp-schedules-21,4a3e36642895f476,421d2c736bab142c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:40:13.087 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,bee7f0610145df36,373a4fe29e2307e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments?taskInstanceId=2&subtaskInstanceId=26, client=10.11.12.102, traceId=bee7f0610145df36, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=2,26,] 2025-04-08 20:40:13.090 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,bee7f0610145df36,373a4fe29e2307e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2/subtasks/26/logSegments client=10.11.12.102, traceId=bee7f0610145df36, duration=7 ms] 2025-04-08 20:40:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:40:13.176769+08:00, lastProcessTime=2025-04-08T20:40:03.177221+08:00 2025-04-08 20:40:13.267 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:40:13.176769+08:00 2025-04-08 20:40:13.834 INFO 3009027 --- [subtask-coordinator3,1308668168866431,3457003810769e73] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=3, original=READY 2025-04-08 20:40:13.836 INFO 3009027 --- [subtask-coordinator3,1308668168866431,3457003810769e73] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 3, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:13.867 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,98dc3a8527eb19a2] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:13.874 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,98dc3a8527eb19a2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=3, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:13.834+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=3}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:13.882 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,98dc3a8527eb19a2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 3, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:13.883 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,98dc3a8527eb19a2] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[25] 2025-04-08 20:40:14.011 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,49f1719ea7942ab1,030be554401f2f24] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=49f1719ea7942ab1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:14.021 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,49f1719ea7942ab1,030be554401f2f24] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=49f1719ea7942ab1, duration=14 ms] 2025-04-08 20:40:14.230 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:14.230 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:14.863 INFO 3009027 --- [subtask-coordinator3,1308668168866431,dc425efd2b288e5d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=25, original=READY 2025-04-08 20:40:14.865 INFO 3009027 --- [subtask-coordinator3,1308668168866431,dc425efd2b288e5d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 25, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:14.979 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,1b121f314423af0d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:14.982 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,1b121f314423af0d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=25, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:14.863+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=25}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:14.986 WARN 3009027 --- [ocp-async-10,94d5f4bbba389e12,9cd7e40ee7fb6ad3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:14.986 INFO 3009027 --- [ocp-async-1,e0c2fc1e96ea295d,5d323ea504cd3fad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:14.986 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,1b121f314423af0d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 25, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:14.988 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,1b121f314423af0d] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[7] 2025-04-08 20:40:15.295 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,36f047f276f9878b,b1190c100a0a9212] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=36f047f276f9878b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:15.304 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,36f047f276f9878b,b1190c100a0a9212] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=36f047f276f9878b, duration=13 ms] 2025-04-08 20:40:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,38e08df5508ac4f1,09c84dd75a183d32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:15.617 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744115940, nodeCount=418, affectRows=418 2025-04-08 20:40:15.619 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744115940, nodeCount=415, validCount=415 2025-04-08 20:40:15.976 INFO 3009027 --- [subtask-coordinator3,1308668168866431,c24c0b28d98aaa31] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=7, original=READY 2025-04-08 20:40:15.978 INFO 3009027 --- [subtask-coordinator3,1308668168866431,c24c0b28d98aaa31] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:16.003 INFO 3009027 --- [ocp-schedules-22,cbb70a5bdbfc5770,22cb3207f46e9526] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:40:16.057 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f085dcdd1b1b3418] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:16.067 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f085dcdd1b1b3418] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=7, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:15.976+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=7}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:16.076 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f085dcdd1b1b3418] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 20:40:16.092 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f085dcdd1b1b3418] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:16.094 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f085dcdd1b1b3418] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[8] 2025-04-08 20:40:16.506 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,e76f0e568df73f5b,59a447f1be06afc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=e76f0e568df73f5b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:16.515 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,e76f0e568df73f5b,59a447f1be06afc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=e76f0e568df73f5b, duration=13 ms] 2025-04-08 20:40:16.986 WARN 3009027 --- [ocp-async-5,9854968c4c1b63d2,fefa9cec3dfeadad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:16.986 INFO 3009027 --- [ocp-async-1,ac30072cbaaedc96,15cf9a94823b7286] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:17.052 INFO 3009027 --- [subtask-coordinator3,1308668168866431,7adccd4d9e7d609c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=8, original=READY 2025-04-08 20:40:17.053 INFO 3009027 --- [subtask-coordinator3,1308668168866431,7adccd4d9e7d609c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 8, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:17.078 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,5fa35b11d3bc8591] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:17.081 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,5fa35b11d3bc8591] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=8, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:17.052+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=8}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:17.086 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,5fa35b11d3bc8591] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 8, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:17.087 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,5fa35b11d3bc8591] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[23] 2025-04-08 20:40:17.230 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:17.231 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:17.757 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,b77f327d241c1899,b16ad0697af961b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=b77f327d241c1899, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:17.766 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,b77f327d241c1899,b16ad0697af961b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=b77f327d241c1899, duration=13 ms] 2025-04-08 20:40:17.894 INFO 3009027 --- [ocp-schedules-27,7acb9662c8f3fc60,08ca1749700e22bc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:40:17.895 INFO 3009027 --- [ocp-schedules-27,7acb9662c8f3fc60,08ca1749700e22bc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:40:18.071 INFO 3009027 --- [subtask-coordinator3,1308668168866431,6414a59adeafe288] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=23, original=READY 2025-04-08 20:40:18.072 INFO 3009027 --- [subtask-coordinator3,1308668168866431,6414a59adeafe288] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 23, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:18.100 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,74c71f0903be4a17] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:18.102 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,74c71f0903be4a17] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=23, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:18.070+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=23}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:18.109 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,74c71f0903be4a17] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 20:40:18.114 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,74c71f0903be4a17] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 23, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:18.115 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,74c71f0903be4a17] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[27] 2025-04-08 20:40:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,494d70c96cc89443,d3cdbd91660cf159] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:18.986 WARN 3009027 --- [ocp-async-9,5c4f41d171046ed8,e82840a20f9b47f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:18.986 INFO 3009027 --- [ocp-async-4,c8704afd920c5639,82f0ac42c66db089] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:18.992 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1be43d1792c690d1,74b70fb34f767325] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=1be43d1792c690d1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:19.003 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,1be43d1792c690d1,74b70fb34f767325] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=1be43d1792c690d1, duration=15 ms] 2025-04-08 20:40:19.095 INFO 3009027 --- [subtask-coordinator3,1308668168866431,5ac6a1eb50b2c076] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=27, original=READY 2025-04-08 20:40:19.097 INFO 3009027 --- [subtask-coordinator3,1308668168866431,5ac6a1eb50b2c076] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 27, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:19.146 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,c4aab9b45c80c39c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:19.165 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,c4aab9b45c80c39c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=27, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:19.095+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=27}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:19.219 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,c4aab9b45c80c39c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 27, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:19.221 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,c4aab9b45c80c39c] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[18] 2025-04-08 20:40:20.103 INFO 3009027 --- [subtask-coordinator3,1308668168866431,749ab8988cfad2f7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=18, original=READY 2025-04-08 20:40:20.104 INFO 3009027 --- [subtask-coordinator3,1308668168866431,749ab8988cfad2f7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 18, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:20.200 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,da1c1a9a7e22bfe1,4d10251022b3fc9e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=da1c1a9a7e22bfe1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:20.125 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,8e98733b7c980b0f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:20.209 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,da1c1a9a7e22bfe1,4d10251022b3fc9e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=da1c1a9a7e22bfe1, duration=13 ms] 2025-04-08 20:40:20.231 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:20.232 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:20.208 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,8e98733b7c980b0f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=18, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:20.102+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=18}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:20.355 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,8e98733b7c980b0f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 18, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:20.357 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,8e98733b7c980b0f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[21] 2025-04-08 20:40:20.986 WARN 3009027 --- [ocp-async-2,43c62a7ae6d420f6,b79be8ce99cec390] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:20.986 INFO 3009027 --- [ocp-async-10,c1558bf8fd185b6c,d43ad282956c76a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:21.117 INFO 3009027 --- [subtask-coordinator3,1308668168866431,c21bda0a8afaabe2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=21, original=READY 2025-04-08 20:40:21.118 INFO 3009027 --- [subtask-coordinator3,1308668168866431,c21bda0a8afaabe2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:21.132 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,7171189357cb74ac] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:21.157 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,7171189357cb74ac] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=21, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:21.116+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=21}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:21.165 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,7171189357cb74ac] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 21, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:21.166 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,7171189357cb74ac] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[20] 2025-04-08 20:40:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,5bdeee048f29bfd3,4ef3ec48b3d8a196] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:21.450 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e8c42e3ded4b1572,7cad3782547cd96d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=e8c42e3ded4b1572, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:21.458 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e8c42e3ded4b1572,7cad3782547cd96d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=e8c42e3ded4b1572, duration=13 ms] 2025-04-08 20:40:22.124 INFO 3009027 --- [subtask-coordinator3,1308668168866431,47f3c1eb4fe3b043] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=20, original=READY 2025-04-08 20:40:22.126 INFO 3009027 --- [subtask-coordinator3,1308668168866431,47f3c1eb4fe3b043] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 20, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:22.214 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:22.217 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=20, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:22.124+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=20}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:22.219 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=1 2025-04-08 20:40:22.223 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1, method=deleteHost 2025-04-08 20:40:22.314 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 1, please retry manually 2025-04-08 20:40:22.363 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=1 2025-04-08 20:40:22.449 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-04-08 20:40:22.454 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1 2025-04-08 20:40:22.462 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 1 2025-04-08 20:40:22.475 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [1] 2025-04-08 20:40:22.484 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [1] 2025-04-08 20:40:22.493 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [1] 2025-04-08 20:40:22.503 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [1] 2025-04-08 20:40:22.518 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [1:192.168.1.61], from status [NEW] to [NEW] 2025-04-08 20:40:22.537 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 20, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:22.538 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,d99f0f2899784aee] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[22] 2025-04-08 20:40:22.676 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,afe3e4838b660067,858ee404256d772e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=afe3e4838b660067, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:22.687 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,afe3e4838b660067,858ee404256d772e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=afe3e4838b660067, duration=15 ms] 2025-04-08 20:40:22.895 INFO 3009027 --- [ocp-schedules-4,8e2c213ea4c0535b,5b47e84be8563ebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:40:22.896 INFO 3009027 --- [ocp-schedules-8,4ee85ab1a1102163,3785b1fdcaf8f698] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:40:22.896 INFO 3009027 --- [ocp-schedules-18,12edbe4aacc6a2a9,1c0e5ad3c22fc55e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:40:22.896 INFO 3009027 --- [ocp-schedules-19,7811011ae0357110,3bb477665d0b691f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:40:22.896 INFO 3009027 --- [ocp-schedules-22,7e52229b6d6e73b1,302290130f2b12e4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:40:22.897 INFO 3009027 --- [ocp-schedules-18,12edbe4aacc6a2a9,1c0e5ad3c22fc55e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:40:22.897 INFO 3009027 --- [ocp-schedules-22,7e52229b6d6e73b1,302290130f2b12e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:40:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:40:22.898 INFO 3009027 --- [ocp-schedules-28,0fbb377d7090aac6,bbe2bc50fb296cbc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:40:22.898 INFO 3009027 --- [ocp-schedules-28,0fbb377d7090aac6,bbe2bc50fb296cbc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:40:22.898 INFO 3009027 --- [ocp-schedules-24,3cb9e186ee343723,2d961102d826fcbe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:40:22.898 INFO 3009027 --- [ocp-schedules-7,8348df41644d15d5,86eac26b0f49d799] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:40:22.898 INFO 3009027 --- [ocp-schedules-24,3cb9e186ee343723,2d961102d826fcbe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:40:22.898 INFO 3009027 --- [ocp-schedules-7,8348df41644d15d5,86eac26b0f49d799] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:40:22.901 INFO 3009027 --- [ocp-schedules-4,8e2c213ea4c0535b,5b47e84be8563ebb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:40:22.903 INFO 3009027 --- [ocp-schedules-19,7811011ae0357110,3bb477665d0b691f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:40:22.914 INFO 3009027 --- [ocp-schedules-10,9c9e3957abfd6258,734a44301d96fcbf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1672,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":418,"seriesIdKeySize":421,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:40:22.914 INFO 3009027 --- [ocp-schedules-10,9c9e3957abfd6258,734a44301d96fcbf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=63, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=63, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:40:22.915 INFO 3009027 --- [ocp-schedules-23,edf0b60ef290b2a7,296b6070c17b3c1b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:40:22.918 INFO 3009027 --- [ocp-schedules-12,97bdd39dd479aea4,818cd9cc82fe71d9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:40:22.919 INFO 3009027 --- [ocp-schedules-12,97bdd39dd479aea4,818cd9cc82fe71d9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:40:22.944 INFO 3009027 --- [ocp-async-6,d95744ab74a454eb,729722a561d9095b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:40:22.962 INFO 3009027 --- [ocp-schedules-29,d8e20fa75af185dd,244fa453a39a36e6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:40:22.966 INFO 3009027 --- [ocp-schedules-20,c515121279abe692,72b2eae4bbb6a0c9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:40:22.966+08:00, clusters=[1] 2025-04-08 20:40:22.966 INFO 3009027 --- [ocp-schedules-20,c515121279abe692,72b2eae4bbb6a0c9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:40:22.982 INFO 3009027 --- [ocp-schedules-2,d1958e773e63ee58,8b978a3968679ace] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:40:22.986 INFO 3009027 --- [ocp-schedules-5,46ed7ff989614890,cf87a952782a7330] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:40:22.986+08:00, clusters=[1] 2025-04-08 20:40:22.986 WARN 3009027 --- [ocp-async-1,301004958495066a,f1a757802ce02c0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:22.986 INFO 3009027 --- [ocp-async-9,68ff797151782025,c53b4ddd997e2c52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:22.988 INFO 3009027 --- [ocp-schedules-5,46ed7ff989614890,cf87a952782a7330] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:40:22.992 INFO 3009027 --- [ocp-schedules-8,0cd3012cb626f3ba,482086f2def48910] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:40:22.992 INFO 3009027 --- [ocp-schedules-8,0cd3012cb626f3ba,482086f2def48910] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:40:22.996 INFO 3009027 --- [ocp-schedules-7,f9d49cd08a4dd9a9,6408ba8bcb107bde] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:40:22.997 INFO 3009027 --- [ocp-schedules-7,f9d49cd08a4dd9a9,6408ba8bcb107bde] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:40:23.000 INFO 3009027 --- [ocp-schedules-19,ccd401e91b418220,84b2cf4b5288ec72] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:40:23.008 ERROR 3009027 --- [ocp-schedules-7,f9d49cd08a4dd9a9,6408ba8bcb107bde] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:40:23.008 INFO 3009027 --- [ocp-schedules-7,f9d49cd08a4dd9a9,6408ba8bcb107bde] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:40:23.009 INFO 3009027 --- [ocp-schedules-26,8dd6d8e8a08e960e,9b2d7cdaffaa525e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:40:23.009 INFO 3009027 --- [ocp-schedules-26,8dd6d8e8a08e960e,9b2d7cdaffaa525e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:40:23.009 INFO 3009027 --- [ocp-schedules-10,90228dc216a26ac3,8262e02a5611f517] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:40:23.010 INFO 3009027 --- [ocp-schedules-10,90228dc216a26ac3,8262e02a5611f517] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:40:23.012 INFO 3009027 --- [ocp-schedules-4,3f8e1cacbf77c59d,df99c39bb25d6a60] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:40:23.013 INFO 3009027 --- [ocp-schedules-4,3f8e1cacbf77c59d,df99c39bb25d6a60] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:40:23.013 INFO 3009027 --- [ocp-schedules-4,3f8e1cacbf77c59d,df99c39bb25d6a60] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:40:23.013 INFO 3009027 --- [ocp-schedules-26,8dd6d8e8a08e960e,9b2d7cdaffaa525e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:40:23.013 INFO 3009027 --- [ocp-schedules-26,8dd6d8e8a08e960e,9b2d7cdaffaa525e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:40:23.033 INFO 3009027 --- [ocp-schedules-12,a243c05ac8e4fdae,3b484e14222d47b0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:40:23.034 INFO 3009027 --- [ocp-schedules-12,a243c05ac8e4fdae,3b484e14222d47b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:40:23.045 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:23.046 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:40:23.056 INFO 3009027 --- [ocp-schedules-16,d38aae8ca49e0530,f289699c0a1adb6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:40:23.056 INFO 3009027 --- [ocp-schedules-1,0f8e7074653486d8,4a9981699c5a4b3e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:40:23.057 INFO 3009027 --- [ocp-schedules-1,0f8e7074653486d8,4a9981699c5a4b3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:40:23.062 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1, status=TAKINGOVER 2025-04-08 20:40:23.062 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:29:47+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:40:23.097 INFO 3009027 --- [ocp-schedules-9,d9595e6f8a46a035,3109933693be6518] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:40:23.107 INFO 3009027 --- [ocp-schedules-16,2188a06f8b7b4864,4679a37618261225] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:40:23.108 INFO 3009027 --- [ocp-schedules-16,2188a06f8b7b4864,4679a37618261225] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1] 2025-04-08 20:40:23.108 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:40:23.108 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:40:23.111 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:40:23.111 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766199115910868992 2025-04-08 20:40:23.114 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=766200751622320128 2025-04-08 20:40:23.114 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:40:23.114 INFO 3009027 --- [ocp-schedules-25,54a7741900ef95b6,490b14d3d8ede566] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:40:23.123 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:40:23.124 ERROR 3009027 --- [ocp-schedules-16,2188a06f8b7b4864,4679a37618261225] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:40:23.124 INFO 3009027 --- [ocp-schedules-16,2188a06f8b7b4864,4679a37618261225] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:40:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:40:23.176632+08:00, lastProcessTime=2025-04-08T20:40:13.176769+08:00 2025-04-08 20:40:23.210 INFO 3009027 --- [subtask-coordinator3,1308668168866431,742a9d2f6ec61052] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=22, original=READY 2025-04-08 20:40:23.211 INFO 3009027 --- [subtask-coordinator3,1308668168866431,742a9d2f6ec61052] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 22, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 20:40:23.232 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:23.232 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:23.225 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:40:23.234 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=22, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=2, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:40:23.210+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=22}, listMap={add_region_ids=[2], server_ids=[1], add_idc_ids=[1], all_host_ids=[1], add_host_ids=[1], host_ids=[1], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 20:40:23.236 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster metadata 2025-04-08 20:40:23.253 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:40:23.176632+08:00 2025-04-08 20:40:23.254 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.o.i.c.l.CredentialServiceListener : Detect cluster been deleted, begin to delete credentials of this cluster, clusterId=1 2025-04-08 20:40:23.267 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.c.CredentialServiceImpl : Deleting credential: 3 2025-04-08 20:40:23.272 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.c.CredentialServiceImpl : Delete credential 3 success 2025-04-08 20:40:23.288 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.o.i.c.l.CredentialServiceListener : Delete credentials success, clusterId=1 2025-04-08 20:40:23.294 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.o.i.c.l.ClusterServiceListener : detect cluster been deleted, begin to delete parameter change and compaction info, clusterId=1 2025-04-08 20:40:23.394 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=2, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:40:23.396 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.o.i.c.l.ClusterServiceListener : detect parameter change and compaction info finished, clusterId=1 2025-04-08 20:40:23.406 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=3, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["ocpdb"]} 2025-04-08 20:40:23.412 ERROR 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1] 2025-04-08 20:40:23.423 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1 at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$358f1847.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:40:23.446 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.o.internal.ObproxyServiceListener : detect cluster been deleted, begin to delete oblinks, obclusterName=ocpdb, obClusterId=1744114127 2025-04-08 20:40:23.453 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=4, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:40:23.455 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.b.i.l.BackupServiceListener : detect cluster been deleted, begin to delete backup strategy, clusterId=1 2025-04-08 20:40:23.468 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.b.i.l.BackupServiceListener : detect backup strategy finished, clusterId=1 2025-04-08 20:40:23.497 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.s.o.OrganizationResourceManager : delete resource for current organization,organizationId=10000000,resourceType=1, resourceId=1 2025-04-08 20:40:23.562 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 20:40:23.563 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 20:40:23.563 INFO 3009027 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 20:40:23.563 WARN 3009027 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster ocpdb not found in config server 2025-04-08 20:40:23.597 INFO 3009027 --- [ocp-schedules-3,a1a309fe399e3111,099fa1f15f5c3c27] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61, current targets ,,,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61 2025-04-08 20:40:23.559 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster finished 2025-04-08 20:40:23.729 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=2, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:40:23.729 INFO 3009027 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Receive invalidate ob version event, clusters:[ocpdb] 2025-04-08 20:40:23.733 INFO 3009027 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Invalidate ob version success, clusters:[ocpdb] 2025-04-08 20:40:23.687 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host started, hostIds=[1] 2025-04-08 20:40:23.776 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=3, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["ocpdb"]} 2025-04-08 20:40:23.776 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 20:40:23.776 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 20:40:23.776 INFO 3009027 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 20:40:23.777 WARN 3009027 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster ocpdb not found in config server 2025-04-08 20:40:23.778 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=4, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:40:23.777 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1, method=deleteHost 2025-04-08 20:40:23.835 ERROR 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1] 2025-04-08 20:40:23.844 WARN 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 1 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1 2025-04-08 20:40:23.845 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host finished 2025-04-08 20:40:23.870 INFO 3009027 --- [ocp-schedules-7,dbe83cf68d4512a7,826e86abc1fe16cf] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 20:40:23.870 INFO 3009027 --- [ocp-schedules-7,dbe83cf68d4512a7,826e86abc1fe16cf] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 20:40:23.887 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,fa9ad12dedc898b0,ca478e866e0c07c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=fa9ad12dedc898b0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:23.897 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,fa9ad12dedc898b0,ca478e866e0c07c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=fa9ad12dedc898b0, duration=14 ms] 2025-04-08 20:40:23.869 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.host.service.IdcServiceImpl : Deleted idc: id=1 2025-04-08 20:40:23.980 INFO 3009027 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=2 2025-04-08 20:40:23.980 INFO 3009027 --- [ForkJoinPool.commonPool-worker-9,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=3 2025-04-08 20:40:23.980 INFO 3009027 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=4 2025-04-08 20:40:23.972 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.host.service.RegionServiceImpl : Deleted region: 2 deault_region 2025-04-08 20:40:23.988 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : rollback finished 2025-04-08 20:40:23.994 INFO 3009027 --- [manual-subtask-executor16,73ac1c39155401d0,f56f7d32ab3a0cfc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 22, operation:ROLLBACK, state: PENDING 2025-04-08 20:40:23.996 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 20:40:23.997 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 20:40:23.998 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 20:40:23.999 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 20:40:24.000 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.001 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 20:40:24.002 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 20:40:24.003 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 20:40:24.004 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 20:40:24.005 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 20:40:24.006 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 20:40:24.007 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 20:40:24.008 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.009 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 20:40:24.010 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 20:40:24.011 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 20:40:24.012 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 20:40:24.013 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:40:24.013 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 20:40:24.013 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 20:40:24.013 INFO 3009027 --- [ocp-schedules-30,368c515ce8e5b84d,2aa4bd71d3c5f0e4] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 20:40:24.094 INFO 3009027 --- [ocp-schedules-26,b1791238f3502680,ea72a0407037547c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:40:24.391 ERROR 3009027 --- [task-coordinator3,,] c.o.o.s.operation.OperationAlarmService : GetClusterOverview, clusterId:1, NotFoundException {} com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1 at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl.nullSafeGetClusterOverview(ClusterDaoManagerImpl.java:94) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$FastClassBySpringCGLIB$$1f10cbd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$EnhancerBySpringCGLIB$$37595865.nullSafeGetClusterOverview() at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:126) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:40:24.391 WARN 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:40:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,68ca429b14cb22ac,0fdca6bc2807744c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:24.702 INFO 3009027 --- [ocp-schedules-9,765a5b115c4a3838,360374b9aa368396] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb 2025-04-08 20:40:24.702 INFO 3009027 --- [ocp-schedules-9,765a5b115c4a3838,360374b9aa368396] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:40:24.702 INFO 3009027 --- [ocp-schedules-9,765a5b115c4a3838,360374b9aa368396] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb 2025-04-08 20:40:24.702 INFO 3009027 --- [ocp-schedules-9,765a5b115c4a3838,360374b9aa368396] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:40:24.702 INFO 3009027 --- [ocp-schedules-9,765a5b115c4a3838,360374b9aa368396] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 20:40:24.986 WARN 3009027 --- [ocp-async-9,c7e032262ef6cf4b,fd4f159f27d4f5ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:24.986 INFO 3009027 --- [ocp-async-3,68384d08848c31e8,b8d4a0645152d395] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:25.059 INFO 3009027 --- [ocp-schedules-24,37cedaef095218bb,95cab2a881891630] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:40:25.142 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a662e2c919cc3794,17091b256afae738] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/2?taskInstanceId=2, client=10.11.12.102, traceId=a662e2c919cc3794, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=2,] 2025-04-08 20:40:25.142 INFO 3009027 --- [ocp-schedules-24,37cedaef095218bb,95cab2a881891630] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:40:25.142 INFO 3009027 --- [ocp-schedules-24,37cedaef095218bb,95cab2a881891630] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:40:25.151 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a662e2c919cc3794,17091b256afae738] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/2 client=10.11.12.102, traceId=a662e2c919cc3794, duration=13 ms] 2025-04-08 20:40:26.024 INFO 3009027 --- [ocp-schedules-15,0a55aba19eb78405,b6932e2170f2cd7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:40:26.232 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:26.233 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:40:26.987 WARN 3009027 --- [ocp-async-4,cb55bd4d73557e47,a002412f56356493] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:26.987 INFO 3009027 --- [ocp-async-5,f221ceac8800df04,ec0929d5972f5001] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,2a5aae907667bea2,a453527f1fcf6205] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:27.992 INFO 3009027 --- [ocp-schedules-27,70beeee513c46943,3119b880936a8afa] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:40:27.993 INFO 3009027 --- [ocp-schedules-27,70beeee513c46943,3119b880936a8afa] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:40:28.987 WARN 3009027 --- [ocp-async-7,9e776bee25112ddf,46e4e87f7fc99dfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:28.987 INFO 3009027 --- [ocp-async-1,e62180a55cecec57,fce07655f49f5ac3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:29.065 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,42d8813fb177a84e,0c5fdfabee6550c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=42d8813fb177a84e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:40:29.065 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,4397f402bdc8636b,7f868323be6df9c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=4397f402bdc8636b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:40:29.065 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,42d8813fb177a84e,0c5fdfabee6550c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=42d8813fb177a84e, duration=3 ms] 2025-04-08 20:40:29.065 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,eda26b34620ac7f1,04dbe9f92bb2e7d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=eda26b34620ac7f1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:40:29.066 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,eda26b34620ac7f1,04dbe9f92bb2e7d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=eda26b34620ac7f1, duration=3 ms] 2025-04-08 20:40:29.066 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,29acca98993eb04c,e00746c0b3091257] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=29acca98993eb04c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:40:29.068 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,4397f402bdc8636b,7f868323be6df9c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=4397f402bdc8636b, duration=6 ms] 2025-04-08 20:40:29.149 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,29acca98993eb04c,e00746c0b3091257] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=29acca98993eb04c, duration=86 ms] 2025-04-08 20:40:29.222 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:29.222 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:29.222 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:29.233 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:29.233 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:29.260 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,976539aed8f86ad5,064f00ad91d4e5c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=976539aed8f86ad5, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:40:29.264 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,976539aed8f86ad5,064f00ad91d4e5c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=976539aed8f86ad5, duration=7 ms] 2025-04-08 20:40:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,bd8a45920406c967,1c9bd18a9c7d4139] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:30.987 WARN 3009027 --- [ocp-async-9,316941961144253e,dbc1c8675619bbb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:30.987 INFO 3009027 --- [ocp-async-10,b509394b9b59a1b8,8d9907bfb2ee0e13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:31.054 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,633ef81362052adf,25c2535c5a74954c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=633ef81362052adf, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:40:31.054 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,e34ac8c9cb6be0ae,addf4bc3a20e4302] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=e34ac8c9cb6be0ae, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:40:31.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,633ef81362052adf,25c2535c5a74954c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=633ef81362052adf, duration=2 ms] 2025-04-08 20:40:31.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,e34ac8c9cb6be0ae,addf4bc3a20e4302] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=e34ac8c9cb6be0ae, duration=3 ms] 2025-04-08 20:40:31.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a509027dcb23f541,2e34cecb6331710e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=a509027dcb23f541, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:40:31.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,a509027dcb23f541,2e34cecb6331710e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=a509027dcb23f541, duration=3 ms] 2025-04-08 20:40:31.200 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:31.201 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:31.201 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:32.234 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:32.234 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:32.897 INFO 3009027 --- [ocp-schedules-18,12edbe4aacc6a2a9,1c0e5ad3c22fc55e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:40:32.898 INFO 3009027 --- [ocp-schedules-22,7e52229b6d6e73b1,302290130f2b12e4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:40:32.944 INFO 3009027 --- [ocp-async-4,3e6e92870d51e2f7,43db95a4fd175ab2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:40:32.987 WARN 3009027 --- [ocp-async-8,056c48dbe7d6ce5a,c0d65201f6871ace] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:32.987 INFO 3009027 --- [ocp-async-7,70e46b85ce28011a,f932eb848c064eb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:33.001 INFO 3009027 --- [ocp-schedules-30,046cc2973eda6513,aa2586b61b2d2cda] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:40:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:40:33.176578+08:00, lastProcessTime=2025-04-08T20:40:23.176632+08:00 2025-04-08 20:40:33.289 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:40:33.176578+08:00 2025-04-08 20:40:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,ad81c8c433e87349,d776c7bd8c5c2ad1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:34.008 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,8429f4be086703d7,7a60fd938656de3a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=8429f4be086703d7, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:40:34.008 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,20258f113ab220b2,86bafd6962af23d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=20258f113ab220b2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:40:34.009 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,f79eeea320c09739,185470896bfc4aa1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=f79eeea320c09739, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:40:34.009 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,acd9673a2da228ea,b9de177283b44c52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=acd9673a2da228ea, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:40:34.009 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,ef547232ba0ede08,e329edeed8532ec3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A40%3A33%2B08%3A00&endTime=2025-04-08T20%3A40%3A33%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=ef547232ba0ede08, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:40:33+08:00,2025-04-08T20:40:33+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:40:34.009 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,acd9673a2da228ea,b9de177283b44c52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=acd9673a2da228ea, duration=2 ms] 2025-04-08 20:40:34.010 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,865d3ce66d932817,3ab5e93a1337985a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=865d3ce66d932817, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:40:34.010 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,865d3ce66d932817,3ab5e93a1337985a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=865d3ce66d932817, duration=3 ms] 2025-04-08 20:40:34.012 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,8429f4be086703d7,7a60fd938656de3a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=8429f4be086703d7, duration=7 ms] 2025-04-08 20:40:34.013 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,f79eeea320c09739,185470896bfc4aa1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=f79eeea320c09739, duration=6 ms] 2025-04-08 20:40:34.014 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,20258f113ab220b2,86bafd6962af23d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=20258f113ab220b2, duration=9 ms] 2025-04-08 20:40:34.026 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,ef547232ba0ede08,e329edeed8532ec3] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 20:40:34.027 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,ef547232ba0ede08,e329edeed8532ec3] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744112445 2025-04-08 20:40:34.028 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,ef547232ba0ede08,e329edeed8532ec3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=ef547232ba0ede08, duration=22 ms] 2025-04-08 20:40:34.067 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,c434a4a49ac6d60a,1ff2d14a997c9eb2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=c434a4a49ac6d60a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:40:34.067 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,c434a4a49ac6d60a,1ff2d14a997c9eb2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=c434a4a49ac6d60a, duration=3 ms] 2025-04-08 20:40:34.152 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,0d2c55d5b7a38bdc,db80cc78aa888b44] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=0d2c55d5b7a38bdc, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:40:34.152 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4aa14266dd21ae59,c3cdca1df881742a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=4aa14266dd21ae59, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:40:34.153 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,0d2c55d5b7a38bdc,db80cc78aa888b44] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=0d2c55d5b7a38bdc, duration=4 ms] 2025-04-08 20:40:34.153 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,ad3d44ded6b908fb,0529b1f39c2abbb8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=ad3d44ded6b908fb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:40:34.154 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5b73855207fea274,f8fcda9b02f62365] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=5b73855207fea274, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:40:34.155 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,4aa14266dd21ae59,c3cdca1df881742a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=4aa14266dd21ae59, duration=5 ms] 2025-04-08 20:40:34.159 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,ad3d44ded6b908fb,0529b1f39c2abbb8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=ad3d44ded6b908fb, duration=10 ms] 2025-04-08 20:40:34.225 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:34.225 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:34.225 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:34.256 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5b73855207fea274,f8fcda9b02f62365] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=5b73855207fea274, duration=107 ms] 2025-04-08 20:40:34.282 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,2664c35f3a1799eb,97898c059687b47f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=2664c35f3a1799eb, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:40:34.285 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,2664c35f3a1799eb,97898c059687b47f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=2664c35f3a1799eb, duration=6 ms] 2025-04-08 20:40:34.987 WARN 3009027 --- [ocp-async-5,9f3998a502219a54,6fdf0f9a19f37657] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:34.987 INFO 3009027 --- [ocp-async-6,b420b5c570b4cc9f,f70a61a3ca2e42c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:35.234 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:35.235 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:35.989 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,f7c96255d2157514,1e231007aaadb789] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=f7c96255d2157514, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:40:35.989 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,d53539d93306e44a,d28289f18b4f15a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=d53539d93306e44a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:40:35.989 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,cd862647574dc142,24bed79a5290ba47] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=cd862647574dc142, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:40:35.989 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d8e1bec82f65b09f,5c3b03ca29fc18f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A40%3A35%2B08%3A00&endTime=2025-04-08T20%3A40%3A35%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=d8e1bec82f65b09f, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:40:35+08:00,2025-04-08T20:40:35+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:40:35.989 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,f7c96255d2157514,1e231007aaadb789] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=f7c96255d2157514, duration=3 ms] 2025-04-08 20:40:35.989 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,cd862647574dc142,24bed79a5290ba47] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=cd862647574dc142, duration=3 ms] 2025-04-08 20:40:35.990 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d8e1bec82f65b09f,5c3b03ca29fc18f6] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 20:40:35.990 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d8e1bec82f65b09f,5c3b03ca29fc18f6] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744112445 2025-04-08 20:40:35.990 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,d53539d93306e44a,d28289f18b4f15a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=d53539d93306e44a, duration=4 ms] 2025-04-08 20:40:35.991 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,d8e1bec82f65b09f,5c3b03ca29fc18f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=d8e1bec82f65b09f, duration=5 ms] 2025-04-08 20:40:35.991 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,be38287e0ca6491a,bf0eeaad1e1e1d51] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=be38287e0ca6491a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:40:35.995 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,be38287e0ca6491a,bf0eeaad1e1e1d51] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=be38287e0ca6491a, duration=9 ms] 2025-04-08 20:40:36.068 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,932f18b3b94d1d5a,4967f0dce2d63a0f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=932f18b3b94d1d5a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:40:36.074 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,932f18b3b94d1d5a,4967f0dce2d63a0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=932f18b3b94d1d5a, duration=68 ms] 2025-04-08 20:40:36.077 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,d87feb17d095055c,3dfbd4c821004d80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=d87feb17d095055c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:40:36.077 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,d87feb17d095055c,3dfbd4c821004d80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=d87feb17d095055c, duration=2 ms] 2025-04-08 20:40:36.078 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,aa8fc57e1cfe9689,8ee1800598f58221] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=aa8fc57e1cfe9689, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:40:36.078 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,aa8fc57e1cfe9689,8ee1800598f58221] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=aa8fc57e1cfe9689, duration=3 ms] 2025-04-08 20:40:36.094 INFO 3009027 --- [ocp-schedules-21,804b2f8db9cf14f0,6b326f5a1bb02db6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:40:36.124 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,1ebcd0ee321b4f6e,71c6a6b39dd0ad55] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=1ebcd0ee321b4f6e, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:40:36.124 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,1ebcd0ee321b4f6e,71c6a6b39dd0ad55] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=1ebcd0ee321b4f6e, duration=44 ms] 2025-04-08 20:40:36.126 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:36.126 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:36.126 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,54d270452ef97418,0b45a002714bd07e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:36.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:36.872 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/cluster, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:36.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:36.987 WARN 3009027 --- [ocp-async-2,ec6c7f101e3f8b3c,b411422f7892013c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:36.987 INFO 3009027 --- [ocp-async-6,131064ce3c681504,cee590ba523f6104] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:37.574 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,5c3df7ed7b3f1ca9,c6f8109e40385428] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.11.12.102, traceId=5c3df7ed7b3f1ca9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-04-08 20:40:37.575 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,5c3df7ed7b3f1ca9,c6f8109e40385428] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.11.12.102, traceId=5c3df7ed7b3f1ca9, duration=4 ms] 2025-04-08 20:40:37.769 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,0bd1c3cede0a5f6d,73496870f141574e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=0bd1c3cede0a5f6d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:40:37.770 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,c09e088d55b4c447,213ab9fa8cd0c21f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=c09e088d55b4c447, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:40:37.770 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,c09e088d55b4c447,213ab9fa8cd0c21f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=c09e088d55b4c447, duration=3 ms] 2025-04-08 20:40:37.773 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,0bd1c3cede0a5f6d,73496870f141574e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=0bd1c3cede0a5f6d, duration=6 ms] 2025-04-08 20:40:37.923 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:37.923 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:37.923 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:37.976 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,9b07173731bd518e,775bf893870be899] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=9b07173731bd518e, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:40:37.980 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,9b07173731bd518e,775bf893870be899] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=9b07173731bd518e, duration=7 ms] 2025-04-08 20:40:38.099 INFO 3009027 --- [ocp-schedules-29,72d89bcf001af7cf,a64e04b1cb5ab33f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:40:38.100 INFO 3009027 --- [ocp-schedules-29,72d89bcf001af7cf,a64e04b1cb5ab33f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:40:38.146 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,5acfcda205f6d712,eea37f51f10f0e1e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=5acfcda205f6d712, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 20:40:38.146 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,a29dfcc99ca409bb,70e4e15cf4196fbc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=a29dfcc99ca409bb, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 20:40:38.146 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,a26f732e123033ee,f941de25c1adb9ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=a26f732e123033ee, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 20:40:38.147 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,a29dfcc99ca409bb,70e4e15cf4196fbc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=a29dfcc99ca409bb, duration=3 ms] 2025-04-08 20:40:38.147 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,1462a6baa3365e03,6a3c78fefd4b8c0b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=1462a6baa3365e03, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 20:40:38.147 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,a26f732e123033ee,f941de25c1adb9ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=a26f732e123033ee, duration=3 ms] 2025-04-08 20:40:38.148 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,3a100f63716b1dee,e33ebabea2bf4ee4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=3a100f63716b1dee, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:40:38.148 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,5acfcda205f6d712,eea37f51f10f0e1e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=5acfcda205f6d712, duration=5 ms] 2025-04-08 20:40:38.151 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,1462a6baa3365e03,6a3c78fefd4b8c0b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=1462a6baa3365e03, duration=7 ms] 2025-04-08 20:40:38.200 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,57e5dd3a6b0797df,5cd464c29474ab5f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=57e5dd3a6b0797df, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 20:40:38.206 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,57e5dd3a6b0797df,5cd464c29474ab5f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=57e5dd3a6b0797df, duration=10 ms] 2025-04-08 20:40:38.234 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,3a100f63716b1dee,e33ebabea2bf4ee4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=3a100f63716b1dee, duration=90 ms] 2025-04-08 20:40:38.235 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:38.235 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:38.296 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,56ea2f601a4e3ad3,cddce9636a7c1ef1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T19%3A40%3A38%2B08%3A00&endTime=2025-04-08T20%3A40%3A38%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=56ea2f601a4e3ad3, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T19:40:38+08:00,2025-04-08T20:40:38+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 20:40:38.297 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,56ea2f601a4e3ad3,cddce9636a7c1ef1] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 20:40:38.297 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-2,56ea2f601a4e3ad3,cddce9636a7c1ef1] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744112445 2025-04-08 20:40:38.298 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,56ea2f601a4e3ad3,cddce9636a7c1ef1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=56ea2f601a4e3ad3, duration=6 ms] 2025-04-08 20:40:38.987 WARN 3009027 --- [ocp-async-4,58c2e78d70460e4c,d13d009001f6b76c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:38.988 INFO 3009027 --- [ocp-async-3,8e886e1ece821fd7,e4447d1a0cc628c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,e66aec5975c949b3,5686a467153a1d11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:39.752 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,8e4d99d610882d8e,b76e5703875eeb73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=8e4d99d610882d8e, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:40:39.752 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,8e4d99d610882d8e,b76e5703875eeb73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=8e4d99d610882d8e, duration=2 ms] 2025-04-08 20:40:39.752 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,bf254e94b399e4ad,645d36b0a5c3d9e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=bf254e94b399e4ad, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:40:39.752 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,bf254e94b399e4ad,645d36b0a5c3d9e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=bf254e94b399e4ad, duration=2 ms] 2025-04-08 20:40:39.752 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,8f64a987752604b5,796f73f465e173b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=8f64a987752604b5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:40:39.753 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,8f64a987752604b5,796f73f465e173b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=8f64a987752604b5, duration=3 ms] 2025-04-08 20:40:39.908 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:39.908 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:39.908 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:40.001 INFO 3009027 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:40:40.002 INFO 3009027 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:40:40.953 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:40.954 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:40:40.954 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:40:40.955 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:40:40.955 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:40:40.956 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 20:40:40.988 WARN 3009027 --- [ocp-async-3,fd7abb224a2ecff9,870f827f611e3337] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:40.988 INFO 3009027 --- [ocp-async-9,dd715fa2ab30d08f,61222ec647b50807] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:41.235 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:41.236 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,aab592ed60956cb6,e2d4ea7404f2146e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:42.567 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,1295775115652bfb,4efa1c4c40e49bff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=1295775115652bfb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:40:42.567 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,1295775115652bfb,4efa1c4c40e49bff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=1295775115652bfb, duration=3 ms] 2025-04-08 20:40:42.749 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,4f1073d9dd1e1d7e,ba8e694eeb2e2447] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=4f1073d9dd1e1d7e, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:40:42.751 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,4f1073d9dd1e1d7e,ba8e694eeb2e2447] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=4f1073d9dd1e1d7e, duration=5 ms] 2025-04-08 20:40:42.944 INFO 3009027 --- [ocp-async-9,090f32f87fcc325e,f687a5c3cd88806b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:40:42.988 WARN 3009027 --- [ocp-async-10,c447bbcd84057089,d45023f7c76d6043] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:42.988 INFO 3009027 --- [ocp-async-6,ef7713e7e2994188,78713bc4c8c6d709] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:43.002 INFO 3009027 --- [ocp-schedules-9,84804f682021643b,2284d5afafce06e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:40:43.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:40:43.175585+08:00, lastProcessTime=2025-04-08T20:40:33.176578+08:00 2025-04-08 20:40:43.308 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,7fed16661907a5b3,10462a9115c7ead2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=7fed16661907a5b3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:40:43.308 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,7fed16661907a5b3,10462a9115c7ead2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=7fed16661907a5b3, duration=3 ms] 2025-04-08 20:40:43.308 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,bf540da35b84e815,2b8b42ee5a16dd51] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=bf540da35b84e815, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:40:43.309 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,bf540da35b84e815,2b8b42ee5a16dd51] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=bf540da35b84e815, duration=3 ms] 2025-04-08 20:40:43.309 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6a50879b7cb3c708,2530d94f769d2536] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=6a50879b7cb3c708, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:40:43.312 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6a50879b7cb3c708,2530d94f769d2536] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=6a50879b7cb3c708, duration=7 ms] 2025-04-08 20:40:43.352 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:40:43.175585+08:00 2025-04-08 20:40:43.401 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,bad5d2ea87b85c1f,2479407246f46827] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=bad5d2ea87b85c1f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:40:43.471 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,2dead56c58c67943,59ae4ad6868ddbee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=2dead56c58c67943, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:40:43.474 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,2dead56c58c67943,59ae4ad6868ddbee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=2dead56c58c67943, duration=5 ms] 2025-04-08 20:40:43.494 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,bad5d2ea87b85c1f,2479407246f46827] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=bad5d2ea87b85c1f, duration=96 ms] 2025-04-08 20:40:43.498 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:43.498 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:43.498 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:44.236 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:44.236 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:44.988 WARN 3009027 --- [ocp-async-7,6adbb246b58f7de8,5774c51d410c57ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:44.988 INFO 3009027 --- [ocp-async-9,3fb6e8ba3e6c91d5,6ce45f28b882a258] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:45.009 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:28:14+08:00, updateTime=2025-04-08T20:40:14+08:00, expireTime=null, isLeader=false} 2025-04-08 20:40:45.010 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:40:45.143 INFO 3009027 --- [ocp-schedules-16,d8915d50d5d6389b,3ea654bcb0a3db88] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:40:45.147 INFO 3009027 --- [ocp-schedules-16,d8915d50d5d6389b,3ea654bcb0a3db88] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:40:45.147 INFO 3009027 --- [ocp-schedules-16,d8915d50d5d6389b,3ea654bcb0a3db88] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:40:45.300 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,2bd9cfc558ecb730,184f72568a7abd8f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=2bd9cfc558ecb730, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:40:45.300 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,8b062235c77378a3,62c8407e5f094aee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=8b062235c77378a3, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:40:45.300 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,93c2d21694d32bda,31b16df43a9341b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=93c2d21694d32bda, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:40:45.300 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,8b062235c77378a3,62c8407e5f094aee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=8b062235c77378a3, duration=2 ms] 2025-04-08 20:40:45.300 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,93c2d21694d32bda,31b16df43a9341b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=93c2d21694d32bda, duration=2 ms] 2025-04-08 20:40:45.300 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,2bd9cfc558ecb730,184f72568a7abd8f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=2bd9cfc558ecb730, duration=2 ms] 2025-04-08 20:40:45.399 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:40:45.399 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:40:45.399 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:40:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,aba6c5f713c65e57,a0cc8e6ab3866645] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:46.116 INFO 3009027 --- [ocp-schedules-22,f57a63a3914c045f,badabe7e11fc3127] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:40:46.988 WARN 3009027 --- [ocp-async-9,4f83ac7999e4c2a1,fb82effaf6c05dae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:46.988 INFO 3009027 --- [ocp-async-5,d423508251460f17,8a75c6cbddfc0043] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:47.237 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:47.237 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:48.108 INFO 3009027 --- [ocp-schedules-4,95690b05e577a52d,c6717ee40472a451] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:40:48.110 INFO 3009027 --- [ocp-schedules-4,95690b05e577a52d,c6717ee40472a451] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:40:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,c6f740dceda2d933,1804147d1ac17d65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:48.800 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,109f9a858ba71ef0,fd6da2640117092a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=109f9a858ba71ef0, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:40:48.802 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,109f9a858ba71ef0,fd6da2640117092a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=109f9a858ba71ef0, duration=4 ms] 2025-04-08 20:40:48.988 WARN 3009027 --- [ocp-async-4,aaf3076c7088a830,8985f5d5b2d0887b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:48.988 INFO 3009027 --- [ocp-async-2,e26dde922f845417,e95a5866cdd57374] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:50.237 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:50.238 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:50.988 WARN 3009027 --- [ocp-async-3,1e682b7badd02456,ddec613a3ba4690a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:50.988 INFO 3009027 --- [ocp-async-8,7b0c7a9c59788357,ecb5e3912a7c6308] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,4644db0975284c28,4c49d874cbd85e85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:52.896 INFO 3009027 --- [ocp-schedules-11,9c99554ef46f851c,8f2dcbf0a062b0e9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:40:52.896 INFO 3009027 --- [ocp-schedules-26,2eb098813ac12356,4b0e508e32b2f19b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:40:52.896 INFO 3009027 --- [ocp-schedules-29,3668cfed11bba64a,a65760ed58027e2a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:40:52.898 INFO 3009027 --- [ocp-schedules-17,08973ab6ed3a100a,b9a1ead603368f38] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:40:52.898 INFO 3009027 --- [ocp-schedules-7,02179171e0bb6398,69046ac2f62e7dab] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:40:52.898 INFO 3009027 --- [ocp-schedules-7,02179171e0bb6398,69046ac2f62e7dab] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:40:52.898 INFO 3009027 --- [ocp-schedules-17,08973ab6ed3a100a,b9a1ead603368f38] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:40:52.899 INFO 3009027 --- [ocp-schedules-18,af4f6d386a3c19d1,6435718c3666bbdd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:40:52.899 INFO 3009027 --- [ocp-schedules-18,af4f6d386a3c19d1,6435718c3666bbdd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:40:52.935 INFO 3009027 --- [ocp-schedules-24,07ca5521d6d42c64,28cfb77360f399c1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:40:52.944 INFO 3009027 --- [ocp-schedules-11,f75b44723a15d8c7,8cdcd5ab2543f687] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:40:52.944 INFO 3009027 --- [ocp-async-8,4f039a93bcbaff68,a9c7ad09562e1f85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:40:52.984 INFO 3009027 --- [ocp-schedules-15,f1e556a0ca8fbcc1,3b5625d2519d3900] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:40:52.989 WARN 3009027 --- [ocp-async-1,f68e898160ca1f33,92344a24ef933fc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:52.989 INFO 3009027 --- [ocp-async-10,3dcdd68a6a709454,1507a1dc00433334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:53.003 INFO 3009027 --- [ocp-schedules-23,24d37ba6783db597,f772db3c12f50774] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:40:53.062 INFO 3009027 --- [ocp-schedules-1,0f8e7074653486d8,4a9981699c5a4b3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:40:53.062 INFO 3009027 --- [ocp-schedules-1,01daf919aa3be32a,f725c8eadaa34cf9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:40:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:40:53.176833+08:00, lastProcessTime=2025-04-08T20:40:43.175585+08:00 2025-04-08 20:40:53.238 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:53.239 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:53.329 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:40:53.176833+08:00 2025-04-08 20:40:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,9d8a92ac3b4911c3,4356945c74665ea7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:54.989 WARN 3009027 --- [ocp-async-4,0f67a9dcdff2fc40,5f975ea839c4c737] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:54.989 INFO 3009027 --- [ocp-async-2,9cfb325eedac1ba4,eb574e9a352cef8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:56.142 INFO 3009027 --- [ocp-schedules-30,e019908f965897a3,4628fc2b457069ec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:40:56.239 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:56.239 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:40:56.989 WARN 3009027 --- [ocp-async-3,9f778ddb3b11da0f,52d349e807420224] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:56.989 INFO 3009027 --- [ocp-async-7,42bb84d27b5c3bc2,c7dd332e380d8476] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,e87ee8aca3a899e2,67e44e1201439c95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116060, interval=60 2025-04-08 20:40:58.117 INFO 3009027 --- [ocp-schedules-30,b275d2ed938cfaa3,88b0c7853a261fb7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:40:58.118 INFO 3009027 --- [ocp-schedules-30,b275d2ed938cfaa3,88b0c7853a261fb7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:40:58.989 WARN 3009027 --- [ocp-async-1,b3b58b59388a6083,ac21b57f4a837b8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:40:58.989 INFO 3009027 --- [ocp-async-9,819064032fac9283,e45be87ded39a7cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:40:59.240 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:40:59.240 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,2679cab5e4966a3b,74920d6be8240301] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:00.989 WARN 3009027 --- [ocp-async-4,e5094482efce1057,a83b9184666563b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:00.989 INFO 3009027 --- [ocp-async-5,dde8b1f658a02ebd,35b614a151570ff7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:02.240 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:02.241 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:02.944 INFO 3009027 --- [ocp-async-5,1e35f50bfd9c523f,188d922df1383589] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:41:02.989 WARN 3009027 --- [ocp-async-7,ada1bd20df96aca5,933f308d76b4de9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:02.989 INFO 3009027 --- [ocp-async-3,2239304e2ba746c9,962e6b54d587f460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:03.004 INFO 3009027 --- [ocp-schedules-13,7af6aaaf398594ed,b0a0471146176488] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:41:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:41:03.177018+08:00, lastProcessTime=2025-04-08T20:40:53.176833+08:00 2025-04-08 20:41:03.265 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:41:03.177018+08:00 2025-04-08 20:41:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,fb6eb9e0d0f10862,8aa3e490a79e1dfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:04.989 WARN 3009027 --- [ocp-async-10,ddcccfd0abbaac81,4a9fe7f1e8301b5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:04.989 INFO 3009027 --- [ocp-async-8,947ee336856577d3,22b232b79086c60d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:05.148 INFO 3009027 --- [ocp-schedules-14,246593439906396b,baf25e4b15fd815e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:41:05.213 INFO 3009027 --- [ocp-schedules-14,246593439906396b,baf25e4b15fd815e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:41:05.213 INFO 3009027 --- [ocp-schedules-14,246593439906396b,baf25e4b15fd815e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:41:05.241 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:05.241 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:05.673 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOverPreCheck, client 10.11.12.102, traceId fb756194379525c6] 2025-04-08 20:41:05.676 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOverPreCheck?null, client=10.11.12.102, traceId=fb756194379525c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(TakeOverClusterPreCheckParam), args=TakeOverClusterPreCheckParam(connectionMode=direct, clusterName=null, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=192.168.1.61, port=2881),] 2025-04-08 20:41:05.685 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.685 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.686 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.686 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.687 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.687 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.688 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.689 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.689 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.690 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.691 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 20:41:05.696 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.697 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.697 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.698 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 20:41:05.703 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.703 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.704 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.705 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 20:41:05.710 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.710 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.711 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.711 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.715 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.716 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 20:41:05.716 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.716 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.717 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.717 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.718 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.718 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 20:41:05.723 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.723 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.727 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.728 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 20:41:05.729 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.729 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.733 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.735 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 20:41:05.736 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.737 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.737 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.738 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.741 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:41:05.742 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.744 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:41:05.749 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.749 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.753 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.753 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:41:05.769 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.770 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.770 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.771 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.772 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.772 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.773 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.773 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.774 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.774 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:41:05.789 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.789 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.793 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.793 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:41:05.795 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.795 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.796 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.796 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:41:05.811 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.811 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.812 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.812 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.813 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.813 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.814 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.815 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.815 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.815 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:41:05.831 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.832 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.833 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.833 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:41:05.834 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.835 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.836 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.836 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:41:05.851 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.851 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.852 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.852 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.853 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.853 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.853 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.854 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.854 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.855 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:41:05.870 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.871 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:41:05.876 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.876 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.877 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.877 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.878 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.878 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.879 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.879 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 20:41:05.899 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.900 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.901 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.901 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.902 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.902 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:41:05.903 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.903 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:41:05.904 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.904 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:41:05.908 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.909 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.909 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.910 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.910 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.911 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.911 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.912 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:05.912 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:05.912 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:41:05.920 ERROR 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 20:41:05.921 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,fb756194379525c6,08d23fd0b5397592] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.11.12.102, traceId=fb756194379525c6, duration=250 ms] 2025-04-08 20:41:06.074 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,ba194e627f3d0a9c,1a57350f0bfb1f5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=ba194e627f3d0a9c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 20:41:06.074 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,637f84849e257df8,7999106635935f1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=637f84849e257df8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 20:41:06.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,637f84849e257df8,7999106635935f1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=637f84849e257df8, duration=3 ms] 2025-04-08 20:41:06.078 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,ba194e627f3d0a9c,1a57350f0bfb1f5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=ba194e627f3d0a9c, duration=7 ms] 2025-04-08 20:41:06.160 INFO 3009027 --- [ocp-schedules-28,221b9f45808b5240,159e2fb01d48c4e2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:41:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,9ae4e939fad481f6,5d833370ebadb95a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:06.989 WARN 3009027 --- [ocp-async-8,6f14fba72a63361e,f182ca37b2cf6878] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:06.989 INFO 3009027 --- [ocp-async-2,161223adbce344fb,6f3c0a8166a02410] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:08.124 INFO 3009027 --- [ocp-schedules-22,feb38e597ebc72d1,d02e9acc9108edb1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:41:08.125 INFO 3009027 --- [ocp-schedules-22,feb38e597ebc72d1,d02e9acc9108edb1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:41:08.242 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:08.242 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:08.990 WARN 3009027 --- [ocp-async-7,73392ca5ac7225ce,25c4557fd52ad034] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:08.990 INFO 3009027 --- [ocp-async-5,5381c50ac054f978,9a0a1c7cad624bcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,c74dfb2b93380d97,3fbcd220ecacc143] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:10.002 INFO 3009027 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:41:10.002 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:41:10.989 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:41:10.990 INFO 3009027 --- [ocp-async-3,367da81e99480e39,b5553f995cd0511b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:10.990 WARN 3009027 --- [ocp-async-10,b9e824b3341b1cb4,ec55169ce9bc3bc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:11.242 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:11.243 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:11.848 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,b6c618e61e9a0dcc,f5831385c38406ee] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId b6c618e61e9a0dcc] 2025-04-08 20:41:11.850 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,b6c618e61e9a0dcc,f5831385c38406ee] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=b6c618e61e9a0dcc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 20:41:11.853 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,b6c618e61e9a0dcc,f5831385c38406ee] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)] 2025-04-08 20:41:11.854 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:11.870 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.870 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:11.884 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.885 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:11.899 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.899 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.899 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:11.914 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.954 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.61 2025-04-08 20:41:11.979 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.980 INFO 3009027 --- [pool-24-thread-1,b6c618e61e9a0dcc,cb2f39281a6533d7] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:11.980 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,b6c618e61e9a0dcc,f5831385c38406ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=b6c618e61e9a0dcc, duration=134 ms] 2025-04-08 20:41:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,d387f70aef964c51,c1ba35c31ee8bfc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:12.408 INFO 3009027 --- [prometheus-cache-monitor1,0f7deb199a94c20d,18da8ecc771646bb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1672,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":422,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":416} 2025-04-08 20:41:12.408 INFO 3009027 --- [prometheus-cache-monitor1,0f7deb199a94c20d,18da8ecc771646bb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:41:12.944 INFO 3009027 --- [ocp-async-3,0618a9b4d4a831cc,b53454cec7e25823] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:41:12.990 WARN 3009027 --- [ocp-async-9,5bafec3fa286d291,813003b069bd39a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:12.990 INFO 3009027 --- [ocp-async-1,080d603e41744c24,63132f7db12ac7c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:13.006 INFO 3009027 --- [ocp-schedules-29,7f08fbd484fb7619,b730c6dd59a7800e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:41:13.113 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:37:12.976, end=2025-04-08T20:39:13.046} 2025-04-08 20:41:13.116 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:41:13.116 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:41:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:41:13.176667+08:00, lastProcessTime=2025-04-08T20:41:03.177018+08:00 2025-04-08 20:41:13.315 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:41:13.176667+08:00 2025-04-08 20:41:14.243 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:14.244 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:14.990 WARN 3009027 --- [ocp-async-3,412e9e1c7640f31f,332649c782d29d69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:14.990 INFO 3009027 --- [ocp-async-2,3b3f84f1599b40d2,107350b05628c1c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:15.041 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:29:44+08:00, updateTime=2025-04-08T20:40:44+08:00, expireTime=null, isLeader=false} 2025-04-08 20:41:15.042 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 20:41:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,7e2a7173770b1d61,f5bdb77565386dbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:15.636 INFO 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116000, nodeCount=422, affectRows=422 2025-04-08 20:41:15.639 INFO 3009027 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116000, nodeCount=419, validCount=419 2025-04-08 20:41:16.184 INFO 3009027 --- [ocp-schedules-22,eeb79d4852dc375b,e671ffba3bf3c095] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:41:16.990 WARN 3009027 --- [ocp-async-4,596ee7dd7920ca92,9a3c1df93efb8e8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:16.990 INFO 3009027 --- [ocp-async-2,0856017830ab9555,1a1dc718b61690cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:17.244 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:17.244 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:18.151 INFO 3009027 --- [ocp-schedules-18,bc08b735f4f99cda,c5e9c81c69f900c1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:41:18.152 INFO 3009027 --- [ocp-schedules-18,bc08b735f4f99cda,c5e9c81c69f900c1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:41:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,f59f5bf6a902f4af,c63fff672676804a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:18.990 WARN 3009027 --- [ocp-async-2,2598b026f803ac90,2157baa936ae2d56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:18.990 INFO 3009027 --- [ocp-async-8,c37d7c20d59057cd,2496197bf4b0025d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:20.245 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:20.245 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:20.991 WARN 3009027 --- [ocp-async-6,366e38c320df68b2,7ccb3fb3bcd80350] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:20.991 INFO 3009027 --- [ocp-async-3,0d7c43271b5ecd94,fda194d12b593a8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,8ae5c88f844ac6f2,26fa845af7387943] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:22.604 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,61fe4bfb595a30cb,5acd3785da8d8edc] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId 61fe4bfb595a30cb] 2025-04-08 20:41:22.606 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,61fe4bfb595a30cb,5acd3785da8d8edc] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=61fe4bfb595a30cb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 20:41:22.610 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,61fe4bfb595a30cb,5acd3785da8d8edc] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)] 2025-04-08 20:41:22.610 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:22.626 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.626 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:22.642 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.682 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:22.697 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.697 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.698 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:22.712 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.712 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.61 2025-04-08 20:41:22.738 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.738 INFO 3009027 --- [pool-25-thread-1,61fe4bfb595a30cb,4b1530b716954473] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:22.738 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,61fe4bfb595a30cb,5acd3785da8d8edc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=61fe4bfb595a30cb, duration=136 ms] 2025-04-08 20:41:22.797 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOver, client 10.11.12.102, traceId 24f1d56478c4d70e] 2025-04-08 20:41:22.799 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOver?null, client=10.11.12.102, traceId=24f1d56478c4d70e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster(TakeOverClusterParam), args=TakeOverClusterParam(switchConfigUrl=true, hostInfo=TakeOverClusterParam.HostInfo(kind=DEDICATED_PHYSICAL_MACHINE, publishPorts=null, hostTypeId=1, sshPort=22, credentialId=1)),] 2025-04-08 20:41:22.806 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-192.168.1.61#2881, method=takeOverCluster 2025-04-08 20:41:22.809 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.809 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.810 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.811 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.811 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.812 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.812 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.812 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.813 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.813 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.814 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.814 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 20:41:22.819 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.820 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.820 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.821 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 20:41:22.826 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.826 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.827 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.827 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 20:41:22.832 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.832 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.833 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.833 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.834 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.834 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 20:41:22.834 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.835 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.835 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.836 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.837 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.837 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 20:41:22.842 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.843 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.844 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.844 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 20:41:22.845 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.845 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.846 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.848 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 20:41:22.849 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.850 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.850 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.851 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.851 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:41:22.851 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.854 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 20:41:22.858 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.859 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.860 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.860 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:41:22.876 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.877 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.877 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.878 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.879 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.879 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.879 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.880 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.880 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.881 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:41:22.890 INFO 3009027 --- [ocp-schedules-14,2610d2ca735f470e,6514dd179d353057] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T20:41:22.890+08:00, tenantSize=0 2025-04-08 20:41:22.895 INFO 3009027 --- [ocp-schedules-28,40841ffefc618480,0b99e2a5d6197254] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:41:22.896 INFO 3009027 --- [ocp-schedules-21,c11961a20d8a75b5,4ddfb854e4b84fcc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:41:22.896 INFO 3009027 --- [ocp-schedules-10,21d2544336b62352,2603d902272d1ecf] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:41:22.896 INFO 3009027 --- [ocp-schedules-4,f480dd51a3a3232d,5d7681c77463adca] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:41:22.896 INFO 3009027 --- [ocp-schedules-17,063d95f2e38d3414,d2bdc3ff529d20be] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:41:22.896 INFO 3009027 --- [ocp-schedules-20,2c47fe94aa282a08,e0770874f0e1e9c6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:41:22.898 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.898 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.898 INFO 3009027 --- [ocp-schedules-3,7ecbae9206d92c0a,d6d167570c7f396b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:41:22.898 INFO 3009027 --- [ocp-schedules-3,7ecbae9206d92c0a,d6d167570c7f396b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:41:22.899 INFO 3009027 --- [ocp-schedules-5,acf18891a968442a,d75d777a670f80c6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:41:22.899 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.899 INFO 3009027 --- [ocp-schedules-5,acf18891a968442a,d75d777a670f80c6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:41:22.899 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 20:41:22.899 INFO 3009027 --- [ocp-schedules-1,8ec51d75c11a62ae,052df11b4c4405f4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:41:22.900 INFO 3009027 --- [ocp-schedules-1,8ec51d75c11a62ae,052df11b4c4405f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:41:22.900 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.900 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.901 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.902 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:41:22.905 INFO 3009027 --- [ocp-schedules-4,f480dd51a3a3232d,5d7681c77463adca] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:41:22.917 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.918 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.919 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.919 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.920 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.920 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.921 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.921 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.922 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.922 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:41:22.938 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.938 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.939 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.939 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 20:41:22.940 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.940 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.941 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.941 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:41:22.945 INFO 3009027 --- [ocp-async-9,7492ef64b1207ca0,c4397c3d07f580cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:41:22.956 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:41:22.956 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:41:22.957 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.958 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.959 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.959 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.960 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.960 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.961 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:41:22.961 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:41:22.961 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:41:22.961 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.961 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.962 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.962 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:41:22.962 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:41:22.962 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:41:22.962 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:41:22.963 INFO 3009027 --- [ocp-schedules-24,cfbce139179d9763,5dadf6b5ec706a1b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:41:22.964 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:41:22.964 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:41:22.964 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:41:22.966 INFO 3009027 --- [ocp-schedules-6,17ff1cd9b2383496,0df42f96ac2daa41] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:41:22.967 INFO 3009027 --- [ocp-schedules-23,7a1a00a20e34f085,335987312b8a1c93] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:41:22.967+08:00, clusters=[] 2025-04-08 20:41:22.978 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.978 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.979 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.979 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 20:41:22.983 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.983 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.984 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.984 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.985 INFO 3009027 --- [ocp-schedules-14,5a3d28a9fd1f27c1,d42d7de7a0209e59] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:41:22.985 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.986 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.986 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.987 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 20:41:22.989 INFO 3009027 --- [ocp-schedules-18,4ec2231d06f8e444,acadbe47b43a7527] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:41:22.989+08:00, clusters=[] 2025-04-08 20:41:22.991 WARN 3009027 --- [ocp-async-7,7e13b106422b175a,fd64a1fdda88495d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:22.991 INFO 3009027 --- [ocp-async-2,1c38a1e2f4048646,de02639d111d98b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:22.994 INFO 3009027 --- [ocp-schedules-17,3f6213063d2a1511,a5053fefff5b8c1f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:41:22.994 INFO 3009027 --- [ocp-schedules-17,3f6213063d2a1511,a5053fefff5b8c1f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:41:22.998 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.998 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:22.999 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:22.999 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.000 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.000 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:41:23.000 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.001 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:41:23.001 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.001 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:41:23.005 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.005 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.006 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.006 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.006 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.007 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.007 INFO 3009027 --- [ocp-schedules-20,142e932600708f53,e6d0ed1452203446] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:41:23.007 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.008 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.008 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.008 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:41:23.012 INFO 3009027 --- [ocp-schedules-21,30616d8959b8b5a8,f99fb29611d09e27] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:41:23.012 INFO 3009027 --- [ocp-schedules-21,30616d8959b8b5a8,f99fb29611d09e27] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:41:23.013 INFO 3009027 --- [ocp-schedules-3,984b00f080c3086f,67a2f4e8495622ae] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:41:23.013 INFO 3009027 --- [ocp-schedules-5,9edc652f35637180,46b19839d84335a2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:41:23.013 INFO 3009027 --- [ocp-schedules-5,9edc652f35637180,46b19839d84335a2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:41:23.014 INFO 3009027 --- [ocp-schedules-3,984b00f080c3086f,67a2f4e8495622ae] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:41:23.014 INFO 3009027 --- [ocp-schedules-3,984b00f080c3086f,67a2f4e8495622ae] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 20:41:23.014 INFO 3009027 --- [ocp-schedules-5,9edc652f35637180,46b19839d84335a2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:41:23.014 INFO 3009027 --- [ocp-schedules-5,9edc652f35637180,46b19839d84335a2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:41:23.015 ERROR 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 20:41:23.017 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.host.service.RegionServiceImpl : Created region: 3 deault_region 2025-04-08 20:41:23.018 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.host.service.IdcServiceImpl : Created idc: 2 default_idc 2025-04-08 20:41:23.026 INFO 3009027 --- [ocp-schedules-28,40841ffefc618480,0b99e2a5d6197254] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:41:23.029 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=2, resourceId=2 2025-04-08 20:41:23.030 INFO 3009027 --- [ocp-schedules-2,d4872be641789524,047318bff4ed3b9a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:41:23.033 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=2, resourceId=2 2025-04-08 20:41:23.042 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=2, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:41:23.045 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=2, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:41:23.045 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=2, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:41:23.047 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.CredentialServiceImpl : Credential clusterName=ocpdb|tenantName=sys|username=root is already exists 2025-04-08 20:41:23.047 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.CredentialServiceImpl : Delete credential:clusterName=ocpdb|tenantName=sys|username=root 2025-04-08 20:41:23.049 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=2, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 20:41:23.049 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.049 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:41:23.050 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.050 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:41:23.051 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.051 INFO 3009027 --- [ocp-schedules-12,a243c05ac8e4fdae,3b484e14222d47b0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:41:23.051 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:41:23.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.055 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.056 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.056 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.057 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.057 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.058 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.058 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:41:23.059 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:41:23.059 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 20:41:23.064 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.o.i.c.ClusterOperationServiceImpl : meta cluster detected. 2025-04-08 20:41:23.074 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE APP GROUP, property:UpdateAppGroupProperty(serialNumbers=[null], appGroup=null) 2025-04-08 20:41:23.074 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=5, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:41:23.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Take over primary OB cluster, nodeCount=39, taskCount=39 2025-04-08 20:41:23.075 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=2, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, createTime=null, startTime=2025-04-08T20:41:23.075+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update host arch info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UpdateHostSystemInfoTask, seriesId=19, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Register config url, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RegisterConfigUrlTask, seriesId=7, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Attach cluster attributes, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.AttachClusterAttributesTask, seriesId=30, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=32, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Create default user, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.busine 2025-04-08 20:41:23.099 INFO 3009027 --- [ocp-schedules-8,5ae889682079cf2b,6b1696095a7ed20a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:41:23.104 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=61} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.112 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=62} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.113 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=68} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.116 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=59} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=2, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.116 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=71} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.117 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:41:23.117 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:41:23.117 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=56} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.117 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:41:23.117 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200751622320128 2025-04-08 20:41:23.118 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=766200877631795200 2025-04-08 20:41:23.118 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:41:23.118 INFO 3009027 --- [ocp-schedules-29,5874434d984e2ca5,e2badbb306397ef3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:41:23.118 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=64} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.119 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=72} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=2, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=2, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.119 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=47} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.120 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=53} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.121 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=66} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=2, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=2, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.122 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=65} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=2, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 20:41:23.122 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=8, taskName=Take over primary OB cluster, argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}} 2025-04-08 20:41:23.124 INFO 3009027 --- [ocp-schedules-9,3b42fcd224b4a063,ebd465a71982afa4] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:41:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:41:23.176427+08:00, lastProcessTime=2025-04-08T20:41:13.176667+08:00 2025-04-08 20:41:23.193 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOver client=10.11.12.102, traceId=24f1d56478c4d70e, duration=398 ms] 2025-04-08 20:41:23.193 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,24f1d56478c4d70e,e1def92430f1ad49] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=8, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T20:41:22.799+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 20:41:23.246 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:23.246 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:23.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 20:41:23.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:41:23.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T20:23:49.986+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 20:41:23.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:41:23.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 20:41:23.279 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:41:23.176427+08:00 2025-04-08 20:41:23.297 INFO 3009027 --- [subtask-coordinator3,1308668168866431,297ffea66b0c3b19] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 60, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:23.309 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,4c6798851b62486b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:23.335 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,4c6798851b62486b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=60, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, service_name=ocpdb:1744114127, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, latest_execution_start_time=2025-04-08T20:41:23.295+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=60}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:23.441 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,4c6798851b62486b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 60, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:23.443 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,4c6798851b62486b] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[71] 2025-04-08 20:41:23.630 INFO 3009027 --- [ocp-schedules-22,a38076f5057edcad,c57b06a88c4460e8] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61, current targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61 2025-04-08 20:41:23.814 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 20:41:23.814 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 20:41:23.814 INFO 3009027 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 20:41:23.816 INFO 3009027 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db success. 2025-04-08 20:41:23.875 INFO 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:41:23.875 INFO 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:41:23.877 INFO 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:41:23.879 WARN 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:41:23.879 INFO 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:41:23.880 WARN 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:41:23.883 WARN 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:41:23.883 ERROR 3009027 --- [ocp-schedules-10,6feeada8fcd2a95d,e5ed4bf2941536c8] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=2 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1788) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1669) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2025-04-08 20:41:23.886 INFO 3009027 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=5, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 20:41:23.987 INFO 3009027 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=5 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 20:41:24.063 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 20:41:24.064 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 20:41:24.065 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 20:41:24.066 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 20:41:24.067 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 20:41:24.068 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 20:41:24.069 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 20:41:24.070 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 20:41:24.071 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 20:41:24.072 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 20:41:24.073 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 20:41:24.074 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 20:41:24.075 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 20:41:24.076 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 20:41:24.077 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 20:41:24.078 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 20:41:24.079 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 20:41:24.080 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 20:41:24.081 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 20:41:24.082 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 20:41:24.082 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 20:41:24.082 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 20:41:24.082 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 20:41:24.082 INFO 3009027 --- [ocp-schedules-17,fbcc7d4da1cabbc6,ac7ac96ea96b45a9] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 20:41:24.141 INFO 3009027 --- [ocp-schedules-21,45d3cedbb78ae131,b0f65c1c2437f0ce] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:41:24.307 INFO 3009027 --- [subtask-coordinator3,1308668168866431,ea3d7e94d5626b14] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 71, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:24.317 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:24.320 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=71, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:24.306+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=71}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:24.335 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:24.359 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,9a45a37f18a89b93,dde68f34cc4c8449] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:24.404 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.61 2025-04-08 20:41:24.447 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:24.508 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.512 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.61 2025-04-08 20:41:24.529 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.536 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.538 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 20:41:24.542 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.61, port=22, loopCount=1, minSuccessCount=1 2025-04-08 20:41:24.570 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:24.662 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.707 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.61 2025-04-08 20:41:24.724 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.740 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:24.743 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[198] 2025-04-08 20:41:24.744 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=198, maxAcceptableDelayMillis=3000 2025-04-08 20:41:24.746 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.61], timeoutMillis=20000 2025-04-08 20:41:24.752 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744116084 0 0 , error= 2025-04-08 20:41:24.754 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 20:41:24.804 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 71, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:24.806 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,f77260e0d22e087b] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[81] 2025-04-08 20:41:24.810 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,147ae50c534ed2fe,de09567c5b1e7858] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=147ae50c534ed2fe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 20:41:24.810 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,8069ef09ed17f8d6,d78a18be067e3515] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=8069ef09ed17f8d6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 20:41:24.810 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,8069ef09ed17f8d6,d78a18be067e3515] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=8069ef09ed17f8d6, duration=2 ms] 2025-04-08 20:41:24.813 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,147ae50c534ed2fe,de09567c5b1e7858] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=147ae50c534ed2fe, duration=5 ms] 2025-04-08 20:41:24.854 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,f2ae6263e122ef5c,97b18471a80d3552] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=f2ae6263e122ef5c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:41:24.855 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,f2ae6263e122ef5c,97b18471a80d3552] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=f2ae6263e122ef5c, duration=3 ms] 2025-04-08 20:41:24.855 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,1a4fc241702e7afe,dfae692c64f49c7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=1a4fc241702e7afe, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 20:41:24.921 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,392794f7b34a1f69,520bc3d75fc5c1b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=392794f7b34a1f69, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 20:41:24.928 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,392794f7b34a1f69,520bc3d75fc5c1b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=392794f7b34a1f69, duration=10 ms] 2025-04-08 20:41:24.940 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,1a4fc241702e7afe,dfae692c64f49c7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=1a4fc241702e7afe, duration=88 ms] 2025-04-08 20:41:24.991 WARN 3009027 --- [ocp-async-2,46e61a48b47c2439,b2eb84b013ce84fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:24.991 INFO 3009027 --- [ocp-async-6,1eb1771ee55506d4,2b40d933d3f5cf64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:24.997 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:41:24.997 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:41:24.997 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:41:25.002 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,01ce0dbce118fadc,6e6fa6935f8ba3a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=01ce0dbce118fadc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:25.010 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,01ce0dbce118fadc,6e6fa6935f8ba3a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=01ce0dbce118fadc, duration=11 ms] 2025-04-08 20:41:25.215 INFO 3009027 --- [ocp-schedules-22,83ece740efcb9ecc,9a54655c8dcc0038] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:41:25.280 INFO 3009027 --- [ocp-schedules-22,83ece740efcb9ecc,9a54655c8dcc0038] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:41:25.280 INFO 3009027 --- [ocp-schedules-22,83ece740efcb9ecc,9a54655c8dcc0038] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:41:25.318 INFO 3009027 --- [subtask-coordinator3,1308668168866431,7bc61bc373e73016] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 81, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:25.477 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,742ce1f586736357] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:25.488 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,742ce1f586736357] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=81, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:25.317+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=81}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:25.499 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,742ce1f586736357] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 81, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:25.500 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,742ce1f586736357] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[47] 2025-04-08 20:41:25.632 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,3a70a5fcc4a5e671,967b25ee5e388252] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/81/logSegments?taskInstanceId=8&subtaskInstanceId=81, client=10.11.12.102, traceId=3a70a5fcc4a5e671, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,81,] 2025-04-08 20:41:25.634 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,3a70a5fcc4a5e671,967b25ee5e388252] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/81/logSegments client=10.11.12.102, traceId=3a70a5fcc4a5e671, duration=6 ms] 2025-04-08 20:41:26.199 INFO 3009027 --- [ocp-schedules-28,855889579a8ee7c2,b056122ca81365c6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:41:26.246 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:26.247 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:26.434 INFO 3009027 --- [subtask-coordinator3,1308668168866431,0e2598914de255f9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 47, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:26.447 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,f78581a52cef25bd,bdec5afc492ff8de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=f78581a52cef25bd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:26.456 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,f78581a52cef25bd,bdec5afc492ff8de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=f78581a52cef25bd, duration=12 ms] 2025-04-08 20:41:26.532 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:26.537 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=47, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:26.432+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=47}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:26.543 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 20:41:26.551 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:26.621 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:26.650 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:26.702 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,4b6e7183298efe84,adae4ea301f1a9b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/81/logSegments?taskInstanceId=8&subtaskInstanceId=81, client=10.11.12.102, traceId=4b6e7183298efe84, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,81,] 2025-04-08 20:41:26.704 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,4b6e7183298efe84,adae4ea301f1a9b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/81/logSegments client=10.11.12.102, traceId=4b6e7183298efe84, duration=5 ms] 2025-04-08 20:41:26.709 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:26.750 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.61 2025-04-08 20:41:26.857 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:26.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,838153dcdb3af8db,d39e3e934b287bb6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=838153dcdb3af8db, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 20:41:26.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,838153dcdb3af8db,d39e3e934b287bb6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=838153dcdb3af8db, duration=2 ms] 2025-04-08 20:41:26.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,78ab04ebfd2c0250,3207a8e0b53c52b3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=78ab04ebfd2c0250, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 20:41:26.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,c107952f52330ae8,0cbb3ee5ea2fc150] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=c107952f52330ae8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 20:41:26.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,78ab04ebfd2c0250,3207a8e0b53c52b3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=78ab04ebfd2c0250, duration=2 ms] 2025-04-08 20:41:26.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,c107952f52330ae8,0cbb3ee5ea2fc150] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=c107952f52330ae8, duration=2 ms] 2025-04-08 20:41:26.864 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:26.895 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 20:41:26.895 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 20:41:26.895 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 20:41:26.896 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 20:41:26.905 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 47, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:26.906 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,e654868530fc8740] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[73] 2025-04-08 20:41:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:41:26.991 WARN 3009027 --- [ocp-async-7,636dd2e7964dc287,b1067e66b46ce522] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:26.991 INFO 3009027 --- [ocp-async-4,a52cbfd99f7d53b1,346ed9c08fe1faa6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,656bed2aae3ccfd1,ac7164187fc17ac6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:27.527 INFO 3009027 --- [subtask-coordinator3,1308668168866431,7f0fae311d8bd308] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 73, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:27.638 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,9dcc1c096059f37c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:27.642 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,9dcc1c096059f37c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=73, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:27.525+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=73}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:27.648 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,9dcc1c096059f37c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 73, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:27.649 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,9dcc1c096059f37c] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[65] 2025-04-08 20:41:27.704 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,c385b24fed970e63,c1cc42eecc8f2d67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=c385b24fed970e63, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:27.713 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,c385b24fed970e63,c1cc42eecc8f2d67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=c385b24fed970e63, duration=13 ms] 2025-04-08 20:41:28.183 INFO 3009027 --- [ocp-schedules-15,9eed3427fb79bfb4,90c859543ee39fe3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:41:28.184 INFO 3009027 --- [ocp-schedules-15,9eed3427fb79bfb4,90c859543ee39fe3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:41:28.594 INFO 3009027 --- [subtask-coordinator3,1308668168866431,75a6e876bbd9a152] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 65, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:28.619 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:28.625 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=65, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:28.593+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=65}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:28.628 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 20:41:28.641 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:28.659 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:28.700 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:28.720 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:28.726 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:28.756 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:28.809 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:28.812 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:28.829 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:28.838 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61 2025-04-08 20:41:28.904 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8d94013d80e80960,a689eb842afc87a7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=8d94013d80e80960, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:28.913 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,8d94013d80e80960,a689eb842afc87a7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=8d94013d80e80960, duration=13 ms] 2025-04-08 20:41:28.991 WARN 3009027 --- [ocp-async-5,342b54ae7c3a6ab6,e12ffe37a1be508a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:28.991 INFO 3009027 --- [ocp-async-8,c5d7165a684ebed9,ab8e7629cb2f489e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:29.203 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:29.212 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:29.247 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:29.248 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:29.214 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 20:41:29.281 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 20:41:29.284 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:29.331 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:29.341 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61 2025-04-08 20:41:29.702 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:29.716 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:29.725 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 20:41:29.735 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 20:41:29.813 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,7c7acd5fbcad48a5,5ed9a44cacb19d56] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=7c7acd5fbcad48a5, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 20:41:29.815 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,7c7acd5fbcad48a5,5ed9a44cacb19d56] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=7c7acd5fbcad48a5, duration=5 ms] 2025-04-08 20:41:29.780 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 20:41:29.829 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 65, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:29.831 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,2d34aa85acadf5e2] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[50] 2025-04-08 20:41:30.106 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,515411923a030aeb,e29964ad64586ed6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=515411923a030aeb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:30.115 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,515411923a030aeb,e29964ad64586ed6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=515411923a030aeb, duration=13 ms] 2025-04-08 20:41:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,788735033b3247de,3dac5abd559b1fce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:30.618 INFO 3009027 --- [subtask-coordinator3,1308668168866431,ca471ee10ab5c7d4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 50, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:30.627 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,c38e829f1407a587] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:30.631 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,c38e829f1407a587] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=50, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:30.616+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=50}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:30.643 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,c38e829f1407a587] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 50, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:30.645 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,c38e829f1407a587] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[62] 2025-04-08 20:41:30.991 WARN 3009027 --- [ocp-async-9,49ef54009552ef3c,16e81e74e8984621] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:30.991 INFO 3009027 --- [ocp-async-2,f2d0f2d5064a1aa3,5ef4f3237c4544f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:31.317 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,21948dcc36198c56,5277f0454f044f27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=21948dcc36198c56, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:31.325 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,21948dcc36198c56,5277f0454f044f27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=21948dcc36198c56, duration=12 ms] 2025-04-08 20:41:31.626 INFO 3009027 --- [subtask-coordinator3,1308668168866431,4744a957bd215d50] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 62, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:31.690 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:31.712 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=62, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:31.624+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=62}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:31.714 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 20:41:31.722 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:31.739 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.744 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:31.762 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.803 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:31.835 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.840 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.844 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:31.861 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.901 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:31.927 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.937 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.941 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:31.958 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:31.998 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:32.026 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.071 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.084 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:32.109 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.150 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:32.248 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:32.249 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:32.186 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.252 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.255 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:32.307 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.347 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:32.374 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.383 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.397 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:32.420 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.461 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:32.499 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.504 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.515 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,74ce17951f4e6b96,af29638d6370daee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=74ce17951f4e6b96, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:32.507 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:32.524 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,74ce17951f4e6b96,af29638d6370daee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=74ce17951f4e6b96, duration=12 ms] 2025-04-08 20:41:32.535 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.650 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:32.739 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.782 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.831 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:32.858 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.898 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:32.927 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.937 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.941 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:32.945 INFO 3009027 --- [ocp-async-2,2b1806db03427794,fbbd988367a31066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:41:32.960 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:32.992 WARN 3009027 --- [ocp-async-1,67078a36efc0cbeb,7c7386cf8528b5c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:32.992 INFO 3009027 --- [ocp-async-3,84d290dbcad65217,bb23ae565121d6bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:33.000 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:33.008 INFO 3009027 --- [ocp-schedules-17,5fb123777797099e,627a6ddd90fcbb49] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:41:33.029 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.032 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.041 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:33.060 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.115 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:41:33.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:41:33.176585+08:00, lastProcessTime=2025-04-08T20:41:23.176427+08:00 2025-04-08 20:41:33.280 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:41:33.176585+08:00 2025-04-08 20:41:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,3610395ac174caae,e0b692e681c80e83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:33.484 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.486 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.488 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:41:33.490 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:41:33.492 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:33.509 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.549 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:33.577 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.609 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.652 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:33.732 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.773 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:33.837 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,65fda4e369b42532,d1a8a774b46bc7f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=65fda4e369b42532, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:33.847 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,65fda4e369b42532,d1a8a774b46bc7f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=65fda4e369b42532, duration=13 ms] 2025-04-08 20:41:33.856 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.882 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.887 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:33.903 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:33.944 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:41:33.969 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:34.096 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:34.100 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 20:41:34.105 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:34.121 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:34.162 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:41:34.539 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:34.549 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:34.552 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:41:34.557 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:41:34.563 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 20:41:34.580 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:34.599 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:34.639 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:41:34.992 WARN 3009027 --- [ocp-async-4,e7e461fde19d0211,a204ff98e90f49aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:34.992 INFO 3009027 --- [ocp-async-8,7d523caaf2b56e12,d685cbb293675e02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:35.002 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:35.006 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:35.009 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 20:41:35.020 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 20:41:35.043 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:35.056 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,7f3cb6567c0fc3af,a74dfa03690a0356] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=7f3cb6567c0fc3af, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:35.064 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,7f3cb6567c0fc3af,a74dfa03690a0356] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=7f3cb6567c0fc3af, duration=11 ms] 2025-04-08 20:41:35.061 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:35.101 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61 2025-04-08 20:41:35.132 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:35.249 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:35.249 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:35.177 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:35.279 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 20:41:35.314 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 62, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:35.316 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,dcf58926f472d96a] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[82] 2025-04-08 20:41:35.651 INFO 3009027 --- [subtask-coordinator3,1308668168866431,0e025bdce3416585] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 82, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:35.659 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,a20fa2e8c0a9b391] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:35.665 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,a20fa2e8c0a9b391] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=82, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:35.650+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=82}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:35.740 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,a20fa2e8c0a9b391] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 82, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:35.741 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,a20fa2e8c0a9b391] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[56] 2025-04-08 20:41:35.835 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,48fdf665bd8b5e57,3b1cbace62507f5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/65/logSegments?taskInstanceId=8&subtaskInstanceId=65, client=10.11.12.102, traceId=48fdf665bd8b5e57, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,65,] 2025-04-08 20:41:35.838 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,48fdf665bd8b5e57,3b1cbace62507f5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/65/logSegments client=10.11.12.102, traceId=48fdf665bd8b5e57, duration=6 ms] 2025-04-08 20:41:36.269 INFO 3009027 --- [ocp-schedules-25,f20e03adea321689,9726f63d12cf8110] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:41:36.281 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,53d5e46e9dfcca2a,9f96b16ca9060bbb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=53d5e46e9dfcca2a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:36.292 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,53d5e46e9dfcca2a,9f96b16ca9060bbb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=53d5e46e9dfcca2a, duration=14 ms] 2025-04-08 20:41:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,45f12fdd729218fd,93ecc8f92f033735] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:36.659 INFO 3009027 --- [subtask-coordinator3,1308668168866431,ed735d16f6c63cd3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 56, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:36.753 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:36.772 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=56, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:36.658+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=56}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:36.784 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=83bc254b-9dcf-45f3-824f-a25cf6a28e04, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=2, authType=OCP-HMACSHA1)) 2025-04-08 20:41:36.798 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=83bc254b-9dcf-45f3-824f-a25cf6a28e04, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=2, authType=OCP-HMACSHA1)) 2025-04-08 20:41:36.801 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:7 2025-04-08 20:41:36.806 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 56, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:36.807 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,35dccc3c9b62f33d] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[58] 2025-04-08 20:41:36.992 WARN 3009027 --- [ocp-async-10,0574852c93aad05a,f1602334518fa993] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:36.992 INFO 3009027 --- [ocp-async-7,b369194bd728b0cd,031d60b31eea757f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:37.476 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,f7ed990e26d1bfe7,969c52cc56d5c692] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=f7ed990e26d1bfe7, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:37.485 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,f7ed990e26d1bfe7,969c52cc56d5c692] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=f7ed990e26d1bfe7, duration=13 ms] 2025-04-08 20:41:37.752 INFO 3009027 --- [subtask-coordinator3,1308668168866431,dd49669985a2bffc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 58, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:37.816 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,16cf75a03d81c0ea] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:37.858 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,16cf75a03d81c0ea] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=58, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:37.751+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=58}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:37.914 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,16cf75a03d81c0ea] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 58, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:41:37.916 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,16cf75a03d81c0ea] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[61] 2025-04-08 20:41:38.236 INFO 3009027 --- [ocp-schedules-17,37946b60ab2e86ba,79a5c6ae58ab02db] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:41:38.237 INFO 3009027 --- [ocp-schedules-17,37946b60ab2e86ba,79a5c6ae58ab02db] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:41:38.249 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:38.250 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:38.672 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,25378d21cf32d93c,2749fc3e2211572e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=25378d21cf32d93c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:38.682 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,25378d21cf32d93c,2749fc3e2211572e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=25378d21cf32d93c, duration=13 ms] 2025-04-08 20:41:38.762 INFO 3009027 --- [subtask-coordinator3,1308668168866431,1007746f4f78c6dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:EXECUTE, state: RUNNING 2025-04-08 20:41:38.788 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:41:38.795 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=61, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:41:38.761+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=61}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:41:38.801 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 20:41:38.805 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:41:38.819 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:41:38.823 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:41:38.833 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:41:38.836 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:41:38.845 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:41:38.855 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:38.874 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:38.916 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:38.947 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:38.950 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:41:38.992 WARN 3009027 --- [ocp-async-2,bc4f52a4fdbadcd6,6ce63b78abe83115] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:38.992 INFO 3009027 --- [ocp-async-1,3677c183e47b1526,864769e64ce65ce8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:38.990 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:38.994 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:39.032 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:39.051 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:39.056 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:41:39.102 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:39.160 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:39.178 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 20:41:39.188 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:41:39.210 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:41:39.213 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 20:41:39.249 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:41:39.252 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:41:39.256 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 20:41:39.266 ERROR 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 20:41:39.273 ERROR 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:41:39.277 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:41:39.290 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,32a32e3f8babd1e4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:EXECUTE, state: FAILED 2025-04-08 20:41:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,6a81d502f34dd3e8,de77d2e973d75885] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:39.861 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,bddda5220911386e,13fb107accac59a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=bddda5220911386e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:39.872 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,bddda5220911386e,13fb107accac59a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=bddda5220911386e, duration=14 ms] 2025-04-08 20:41:40.001 INFO 3009027 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:41:40.002 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:41:40.354 INFO 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=8, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=2, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, createTime=2025-04-08T20:41:23+08:00, startTime=2025-04-08T20:41:23.075+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=76, name=Update OB cluster status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.obops.internal.cluster.task.UpdateObClusterStatusTask, seriesId=9, startTime=null, endTime=null}, SubtaskInstanceEntity{id=81, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=18, startTime=2025-04-08T20:41:25.317+08:00, endTime=2025-04-08T20:41:25.493+08:00}, SubtaskInstanceEntity{id=46, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=66, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=34, startTime=null, endTime=null}, SubtaskInstanceEntity{id=68, name=Active agent modules, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obagent.ActiveAgentModuleTask, seriesId=24, startTime=null, endTime=null}, SubtaskInstanceEntity{id=84, name=Wait node, state=PENDING, operation=EXECUTE, 2025-04-08 20:41:40.359 WARN 3009027 --- [task-coordinator3,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:41:40.360 INFO 3009027 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8, status:FAILED 2025-04-08 20:41:40.992 WARN 3009027 --- [ocp-async-4,f24a7706c2bd6b43,60f63e7c4ea313f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:40.992 INFO 3009027 --- [ocp-async-3,675e427210b1b70e,7da80601e9e15950] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:41.128 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,3a1740c43ba2de4d,5e691708a604cac1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=3a1740c43ba2de4d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:41:41.137 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,3a1740c43ba2de4d,5e691708a604cac1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=3a1740c43ba2de4d, duration=14 ms] 2025-04-08 20:41:41.250 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:41.251 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,4b2b6de9633e8b96,97c8a8e9f74e1bb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:42.945 INFO 3009027 --- [ocp-async-3,1ad8b4343cb1969c,5136b9fe03e7080f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:41:42.992 WARN 3009027 --- [ocp-async-6,a0ff8eed674d4ea0,e5951f525b13efa1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:42.992 INFO 3009027 --- [ocp-async-10,4f1cd447fb4ff760,6c2cf9b3ca32b253] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:43.009 INFO 3009027 --- [ocp-schedules-12,43d96053739504a7,c3f2fc44cd6f97cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:41:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:41:43.176802+08:00, lastProcessTime=2025-04-08T20:41:33.176585+08:00 2025-04-08 20:41:43.279 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:41:43.176802+08:00 2025-04-08 20:41:43.749 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,13dc879ca1fe0ce3,381ffa732bb0b7a7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=13dc879ca1fe0ce3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 20:41:43.752 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,13dc879ca1fe0ce3,381ffa732bb0b7a7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=13dc879ca1fe0ce3, duration=6 ms] 2025-04-08 20:41:44.251 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:44.251 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:44.992 WARN 3009027 --- [ocp-async-3,5bcb01fafad22bde,55362bb9dfd7c08d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:44.992 INFO 3009027 --- [ocp-async-1,afa60135e6f53d55,0151f04d77f7665c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:45.281 INFO 3009027 --- [ocp-schedules-29,ad9c71f53c5fc350,165a484d5fbf9534] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:41:45.387 INFO 3009027 --- [ocp-schedules-29,ad9c71f53c5fc350,165a484d5fbf9534] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:41:45.387 INFO 3009027 --- [ocp-schedules-29,ad9c71f53c5fc350,165a484d5fbf9534] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:41:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,b8bf9b0a5926796b,ee9f1458df579e9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:46.290 INFO 3009027 --- [ocp-schedules-7,ffc12cd1439bfe8f,d1b0a2431dfc014c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:41:46.993 WARN 3009027 --- [ocp-async-4,2fa4819ea68e026b,35e1fa3d83bf0d7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:46.993 INFO 3009027 --- [ocp-async-9,308dcf5cf94c5e57,7a69805fdb6194fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:47.251 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:47.252 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:48.271 INFO 3009027 --- [ocp-schedules-17,be95f281eae2e7fd,f356ce0e3a50662e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:41:48.272 INFO 3009027 --- [ocp-schedules-17,be95f281eae2e7fd,f356ce0e3a50662e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:41:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,d43203bde264b1f4,00887d0d466859c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:48.993 WARN 3009027 --- [ocp-async-6,c88e7643cbc34e25,6170261733e308d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:48.993 INFO 3009027 --- [ocp-async-2,53dbf1aae1972e28,21768280d72bb8aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:50.252 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:50.253 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:50.993 WARN 3009027 --- [ocp-async-10,6c6995c4fe703ff2,e8fca9abba56806e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:50.993 INFO 3009027 --- [ocp-async-5,97f7f0d6c0c12552,e1637cf5c7359172] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,f4fd1919520f1fe6,8a594106e1ffeb77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:52.896 INFO 3009027 --- [ocp-schedules-30,8bc1f25d85ac132e,5054386e5449c9e1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:41:52.896 INFO 3009027 --- [ocp-schedules-8,777c797726935316,c6a37811be4c313f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:41:52.896 INFO 3009027 --- [ocp-schedules-20,e9b6060a780f782b,eb0864b271b19c54] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:41:52.896 INFO 3009027 --- [ocp-schedules-8,777c797726935316,c6a37811be4c313f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:41:52.896 INFO 3009027 --- [ocp-schedules-4,2748eced3351ca29,75f55f129760e165] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:41:52.897 INFO 3009027 --- [ocp-schedules-20,e9b6060a780f782b,eb0864b271b19c54] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:41:52.897 INFO 3009027 --- [ocp-schedules-4,2748eced3351ca29,75f55f129760e165] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:41:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:41:52.899 INFO 3009027 --- [ocp-schedules-23,5b86c89078fcf007,4faff544ee0273cc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:41:52.899 INFO 3009027 --- [ocp-schedules-23,5b86c89078fcf007,4faff544ee0273cc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:41:52.899 INFO 3009027 --- [ocp-schedules-9,cd316f2db008abaf,55206cc2b146c994] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:41:52.899 INFO 3009027 --- [ocp-schedules-9,cd316f2db008abaf,55206cc2b146c994] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:41:52.899 INFO 3009027 --- [ocp-schedules-16,f8db2e206c6e3114,a42741e0a57f7bd7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:41:52.899 INFO 3009027 --- [ocp-schedules-16,f8db2e206c6e3114,a42741e0a57f7bd7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:41:52.903 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:41:52.903 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:41:52.927 INFO 3009027 --- [ocp-schedules-25,109d5f6af78b648e,1905e855064ffe26] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:41:52.945 INFO 3009027 --- [ocp-schedules-13,80f2641bb359d67f,e782dd56318734f7] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:41:52.945 INFO 3009027 --- [ocp-async-5,752fd3ffdb49072f,0082f38cd83e44be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:41:52.946 INFO 3009027 --- [ocp-schedules-13,80f2641bb359d67f,e782dd56318734f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:41:52.946 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:41:52.986 INFO 3009027 --- [ocp-schedules-6,f0e7a8f79fb19891,c2e1e6d81efa64b9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:41:52.993 INFO 3009027 --- [ocp-async-8,2dbf2f675645a6b6,62ef53e422b107e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:52.993 WARN 3009027 --- [ocp-async-9,2e8e00abcbd91671,28679228c24597df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:53.011 INFO 3009027 --- [ocp-schedules-27,17f7eb4c73707577,41d95c324844cb09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:41:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:41:53.176227+08:00, lastProcessTime=2025-04-08T20:41:43.176802+08:00 2025-04-08 20:41:53.253 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:53.253 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:53.275 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:41:53.176227+08:00 2025-04-08 20:41:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,bfbd8afc62957d82,174b18a0afb70948] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:54.993 INFO 3009027 --- [ocp-async-6,c76996181a40b62d,16f375086789be76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:54.993 WARN 3009027 --- [ocp-async-7,974ab159ce3d0c61,0b7e01ca47f9236f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:56.253 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:56.254 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:41:56.323 INFO 3009027 --- [ocp-schedules-11,c4aa34b83d0b61b8,abe74d6fda928721] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:41:56.993 WARN 3009027 --- [ocp-async-10,538243d43c6815fb,48b516a034e453a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:56.993 INFO 3009027 --- [ocp-async-6,2728758be602530c,9fb680c24f441123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,fffb68a9276efa3c,672dea20b1c42de0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116120, interval=60 2025-04-08 20:41:58.397 INFO 3009027 --- [ocp-schedules-5,d6561c0b39cb419e,55970d3b089290bb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:41:58.398 INFO 3009027 --- [ocp-schedules-5,d6561c0b39cb419e,55970d3b089290bb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:41:58.993 WARN 3009027 --- [ocp-async-4,f8fada61074c8b4e,7b70983103ef8843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:41:58.993 INFO 3009027 --- [ocp-async-6,8470f5333ef94460,6dfcb9d2c77e695e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:41:59.254 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:41:59.255 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,f43293cd720a7fee,0657b81a75fa5b3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:00.994 INFO 3009027 --- [ocp-async-6,aea0ab0edfd4164d,fdd2e7c54b9131c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:00.994 WARN 3009027 --- [ocp-async-7,a3a79fcb78cdd426,a2978de04a4b8615] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:02.255 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:02.255 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:02.897 INFO 3009027 --- [ocp-schedules-8,777c797726935316,c6a37811be4c313f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:42:02.898 INFO 3009027 --- [ocp-schedules-20,e9b6060a780f782b,eb0864b271b19c54] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:42:02.945 INFO 3009027 --- [ocp-async-6,ff308970f56ba233,3c750e1d0bfe229b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:42:02.946 INFO 3009027 --- [ocp-schedules-13,80f2641bb359d67f,e782dd56318734f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:42:02.994 WARN 3009027 --- [ocp-async-10,a35a42e3dca83ea8,d61283db5e56c251] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:02.994 INFO 3009027 --- [ocp-async-1,fc9031f78d14cb60,d1682ddcb6d4f43f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:03.012 INFO 3009027 --- [ocp-schedules-10,ae9ad9af27de2718,fe754d17de96dc1b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:42:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:42:03.176975+08:00, lastProcessTime=2025-04-08T20:41:53.176227+08:00 2025-04-08 20:42:03.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:42:03.176975+08:00 2025-04-08 20:42:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,e78141d9edfb585d,b6dd2815725325cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:04.994 WARN 3009027 --- [ocp-async-5,8393870a784989eb,92dd0f1bee2d63b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:04.994 INFO 3009027 --- [ocp-async-1,d0d082d9cfd8937c,aeb9d017c3c66d98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:05.256 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:05.256 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:05.389 INFO 3009027 --- [ocp-schedules-3,4a2673fd4ef97e18,fbd11ca6bd2fc46e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:42:05.416 INFO 3009027 --- [ocp-schedules-3,4a2673fd4ef97e18,fbd11ca6bd2fc46e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:42:05.416 INFO 3009027 --- [ocp-schedules-3,4a2673fd4ef97e18,fbd11ca6bd2fc46e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:42:06.344 INFO 3009027 --- [ocp-schedules-23,4de846abb35b8943,3f6c1f89c2f05c15] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:42:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,270fc7f2a4711248,a4d9b3fae3445a12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:06.994 WARN 3009027 --- [ocp-async-1,d4dcc0e1f166603d,2b8450b051154524] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:06.994 INFO 3009027 --- [ocp-async-8,1158c20b533b6123,3add5cdae541dd77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:08.256 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:08.257 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:08.424 INFO 3009027 --- [ocp-schedules-27,abbba53154ba3013,297f291ea60e4373] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:42:08.425 INFO 3009027 --- [ocp-schedules-27,abbba53154ba3013,297f291ea60e4373] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:42:08.994 WARN 3009027 --- [ocp-async-8,2590a5c8b2623571,2da04ea688fe6988] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:08.994 INFO 3009027 --- [ocp-async-10,2a7b1dfb09202c1d,cd3f217ab36f31da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,659a33fe150f3f13,f17e408bef1ac15a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:10.001 INFO 3009027 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:42:10.001 INFO 3009027 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:42:10.989 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:42:10.994 WARN 3009027 --- [ocp-async-7,c580c792d734ada5,12a34b9a144b3d4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:10.994 INFO 3009027 --- [ocp-async-5,657664dbeacec936,3c164b99a6186ba2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:11.257 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:11.257 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,9decb87debcad190,b71f6db10d3ffc26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:12.408 INFO 3009027 --- [prometheus-cache-monitor1,f97dd2391f68dd8d,ddb9236bd36dd3c6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1676,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":422,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:42:12.408 INFO 3009027 --- [prometheus-cache-monitor1,f97dd2391f68dd8d,ddb9236bd36dd3c6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:42:12.945 INFO 3009027 --- [ocp-async-5,b443a9bf41fdd0dd,561f4ddbcdc11f8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:42:12.994 WARN 3009027 --- [ocp-async-2,94312a698847728a,5bd4d8ce97da12cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:12.994 INFO 3009027 --- [ocp-async-1,b6a6d65e20d05a2a,05a8e55fc0e110eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:13.013 INFO 3009027 --- [ocp-schedules-15,b8359523465966b2,ce89c08c360cbf37] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:42:13.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:42:13.177166+08:00, lastProcessTime=2025-04-08T20:42:03.176975+08:00 2025-04-08 20:42:13.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:42:13.177166+08:00 2025-04-08 20:42:14.258 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:14.258 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:14.995 WARN 3009027 --- [ocp-async-1,249a3d53728ec678,56be71707177de7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:14.995 INFO 3009027 --- [ocp-async-4,3496cc105b156f22,80f9ac90019cca62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,fa7aaecae302df90,79fe785954b5a68e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:15.633 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116060, nodeCount=53, affectRows=53 2025-04-08 20:42:15.633 INFO 3009027 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116060, nodeCount=53, validCount=53 2025-04-08 20:42:15.641 INFO 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116060, nodeCount=129, affectRows=129 2025-04-08 20:42:15.641 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116060, nodeCount=126, affectRows=126 2025-04-08 20:42:15.641 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116060, nodeCount=64, affectRows=64 2025-04-08 20:42:15.641 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116060, nodeCount=50, affectRows=50 2025-04-08 20:42:15.641 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116060, nodeCount=64, validCount=64 2025-04-08 20:42:15.641 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116060, nodeCount=50, validCount=50 2025-04-08 20:42:15.642 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116060, nodeCount=125, validCount=125 2025-04-08 20:42:15.642 INFO 3009027 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116060, nodeCount=127, validCount=127 2025-04-08 20:42:16.388 INFO 3009027 --- [ocp-schedules-2,66cdcf3d2314b975,eec931c9f0749891] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:42:16.995 WARN 3009027 --- [ocp-async-9,1d67e9a485349f92,f7828f01bcf2c5b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:16.995 INFO 3009027 --- [ocp-async-4,533e838909132d64,996074d4de1e7620] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:17.258 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:17.259 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,c02c321a1d95c047,cf4b66071a45b44e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:18.435 INFO 3009027 --- [ocp-schedules-6,fe82b7b00cf5a19b,c635876bfdca70fb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:42:18.436 INFO 3009027 --- [ocp-schedules-6,fe82b7b00cf5a19b,c635876bfdca70fb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:42:18.995 WARN 3009027 --- [ocp-async-4,6631451f9b87899f,44c2854d6d37275b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:18.995 INFO 3009027 --- [ocp-async-2,09d688c608d6b916,04b5501a8cd2cddd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:20.259 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:20.260 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:20.995 WARN 3009027 --- [ocp-async-2,a0f7981cea541674,fc3853a4d530e8bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:20.995 INFO 3009027 --- [ocp-async-7,8d083ccd0b9280bf,734e786fbb2856b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,1edccdeca99e1dbf,88975d51835e55a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:22.895 INFO 3009027 --- [ocp-schedules-6,4e1f08d838ca2135,47359bcd2d42dc98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:42:22.896 INFO 3009027 --- [ocp-schedules-20,ad7ddf8956f6d00e,8c393f430fd7e9c7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:42:22.896 INFO 3009027 --- [ocp-schedules-2,c386c31cdcf06a19,82981849d6d1d143] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:42:22.896 INFO 3009027 --- [ocp-schedules-28,5ac02137d8f66e90,15a3361ac828fe89] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:42:22.897 INFO 3009027 --- [ocp-schedules-13,c9e87e77c5480419,28f8b635137800e5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:42:22.897 INFO 3009027 --- [ocp-schedules-2,c386c31cdcf06a19,82981849d6d1d143] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:22.897 INFO 3009027 --- [ocp-schedules-13,c9e87e77c5480419,28f8b635137800e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:42:22.898 INFO 3009027 --- [ocp-schedules-5,ab1c880192b22ca9,1055447fa1551560] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:42:22.898 INFO 3009027 --- [ocp-schedules-16,dd9cf547c672af61,fb05bc2834af6061] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:42:22.898 INFO 3009027 --- [ocp-schedules-5,ab1c880192b22ca9,1055447fa1551560] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:42:22.898 INFO 3009027 --- [ocp-schedules-16,dd9cf547c672af61,fb05bc2834af6061] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:42:22.899 INFO 3009027 --- [ocp-schedules-7,ca9e5a17c8ed2b4f,9b0bc00cc0d07953] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:42:22.899 INFO 3009027 --- [ocp-schedules-7,ca9e5a17c8ed2b4f,9b0bc00cc0d07953] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:42:22.903 INFO 3009027 --- [ocp-schedules-4,2748eced3351ca29,75f55f129760e165] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:42:22.903 INFO 3009027 --- [ocp-schedules-28,5ac02137d8f66e90,15a3361ac828fe89] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:42:22.903 INFO 3009027 --- [ocp-schedules-4,33df6f1e566aef28,58b765e28b451456] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:42:22.904 INFO 3009027 --- [ocp-schedules-4,33df6f1e566aef28,58b765e28b451456] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:22.908 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:42:22.908 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:42:22.914 INFO 3009027 --- [ocp-schedules-22,bcbc751e3cbd410c,d8fd80867493991b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1680,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":420,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:42:22.914 INFO 3009027 --- [ocp-schedules-22,bcbc751e3cbd410c,d8fd80867493991b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:42:22.945 INFO 3009027 --- [ocp-async-5,6549b69d6cc0695c,bcb9786f22459458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:42:22.964 INFO 3009027 --- [ocp-schedules-24,6dd60b34c1c07c96,92ffc74e3270e1a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:42:22.971 INFO 3009027 --- [ocp-schedules-18,c85a517817005e41,404d670cb57b9ad1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:42:22.971+08:00, clusters=[2] 2025-04-08 20:42:22.972 INFO 3009027 --- [ocp-schedules-18,c85a517817005e41,404d670cb57b9ad1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:42:22.987 INFO 3009027 --- [ocp-schedules-17,399bcc15f9c7bfe5,96d6b32b5ee31f58] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:42:22.987 INFO 3009027 --- [ocp-schedules-6,4e1f08d838ca2135,47359bcd2d42dc98] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:42:22.988 INFO 3009027 --- [ocp-schedules-25,d5e67520bf0fa389,2c230bbb293a37fd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:42:22.994 INFO 3009027 --- [ocp-schedules-11,4b740d3ce985301a,e46977080fc47835] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:42:22.994+08:00, clusters=[2] 2025-04-08 20:42:22.995 WARN 3009027 --- [ocp-async-8,284396f07d504789,55adc15885b2fb44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:22.995 INFO 3009027 --- [ocp-async-4,9a87de3fe8a4db0f,780662514e3a749c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:22.995 INFO 3009027 --- [ocp-schedules-11,4b740d3ce985301a,e46977080fc47835] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:42:22.996 INFO 3009027 --- [ocp-schedules-16,1f8be563f37a93e2,1fa9dcecf57edfd6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:42:22.996 INFO 3009027 --- [ocp-schedules-16,1f8be563f37a93e2,1fa9dcecf57edfd6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:42:23.009 INFO 3009027 --- [ocp-schedules-7,bd40b2907839cbb7,b575b3a33da80f97] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:42:23.009 INFO 3009027 --- [ocp-schedules-7,bd40b2907839cbb7,b575b3a33da80f97] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:23.013 INFO 3009027 --- [ocp-schedules-28,7f3be59e44213eff,8afcb5922768c278] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:42:23.013 INFO 3009027 --- [ocp-schedules-28,7f3be59e44213eff,8afcb5922768c278] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:42:23.014 INFO 3009027 --- [ocp-schedules-19,2b1b7ec4236b2ae1,0c058ee537da3bbf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:42:23.015 INFO 3009027 --- [ocp-schedules-22,01f6cd1c246c3a34,2874a6d7d350d123] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:42:23.015 INFO 3009027 --- [ocp-schedules-22,01f6cd1c246c3a34,2874a6d7d350d123] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:42:23.015 INFO 3009027 --- [ocp-schedules-10,e9b3859b2a69f8ea,06350a99fc49446d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:42:23.016 INFO 3009027 --- [ocp-schedules-19,2b1b7ec4236b2ae1,0c058ee537da3bbf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:42:23.016 INFO 3009027 --- [ocp-schedules-19,2b1b7ec4236b2ae1,0c058ee537da3bbf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:42:23.019 INFO 3009027 --- [ocp-schedules-22,01f6cd1c246c3a34,2874a6d7d350d123] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:42:23.019 INFO 3009027 --- [ocp-schedules-22,01f6cd1c246c3a34,2874a6d7d350d123] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:42:23.020 ERROR 3009027 --- [ocp-schedules-7,bd40b2907839cbb7,b575b3a33da80f97] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:42:23.020 INFO 3009027 --- [ocp-schedules-7,bd40b2907839cbb7,b575b3a33da80f97] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:42:23.051 INFO 3009027 --- [ocp-schedules-23,da42ee687e728555,81d32fb2e2983ac0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:42:23.052 INFO 3009027 --- [ocp-schedules-23,da42ee687e728555,81d32fb2e2983ac0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:23.060 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:42:23.061 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:42:23.062 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:42:23.062 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:42:23.104 INFO 3009027 --- [ocp-schedules-27,01a6ec39431616d7,af1314df0eec2221] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:42:23.118 INFO 3009027 --- [ocp-schedules-9,430b306a1c8fbf9d,a0e551ee91f4aa2c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:42:23.118 INFO 3009027 --- [ocp-schedules-9,430b306a1c8fbf9d,a0e551ee91f4aa2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:42:23.118 INFO 3009027 --- [ocp-schedules-9,430b306a1c8fbf9d,a0e551ee91f4aa2c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:42:23.118 INFO 3009027 --- [ocp-schedules-9,430b306a1c8fbf9d,a0e551ee91f4aa2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:42:23.119 INFO 3009027 --- [ocp-schedules-9,430b306a1c8fbf9d,a0e551ee91f4aa2c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:42:23.119 INFO 3009027 --- [ocp-schedules-9,430b306a1c8fbf9d,a0e551ee91f4aa2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:42:23.125 INFO 3009027 --- [ocp-schedules-21,fa84ad263608f947,a8cee77ddc7d6eba] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:42:23.125 INFO 3009027 --- [ocp-schedules-21,fa84ad263608f947,a8cee77ddc7d6eba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:23.141 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:42:23.143 ERROR 3009027 --- [ocp-schedules-21,fa84ad263608f947,a8cee77ddc7d6eba] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:42:23.143 INFO 3009027 --- [ocp-schedules-21,fa84ad263608f947,a8cee77ddc7d6eba] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:42:23.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:42:23.178055+08:00, lastProcessTime=2025-04-08T20:42:13.177166+08:00 2025-04-08 20:42:23.260 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:23.260 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:23.261 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:42:23.178055+08:00 2025-04-08 20:42:23.887 INFO 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:42:23.887 INFO 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:42:23.889 INFO 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:42:23.890 WARN 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:42:23.891 INFO 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:42:23.892 WARN 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:42:23.900 INFO 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.61, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:3 2025-04-08 20:42:23.900 INFO 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:42:23.930 WARN 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:42:23.931 ERROR 3009027 --- [ocp-schedules-11,1456bb7a8bdec5d8,e27e1918529514e9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:42:24.146 INFO 3009027 --- [ocp-schedules-15,bed714f0ea2cca9e,1e223e117d68fc7a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:42:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,21b8a62ceea3e4f1,fdfbf8605b206b83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:24.995 WARN 3009027 --- [ocp-async-10,563d2aba8237d224,c6bb41d5a0f12600] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:24.995 INFO 3009027 --- [ocp-async-4,ccefd2d833f99aab,204562cd862d2a09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:25.418 INFO 3009027 --- [ocp-schedules-11,6a568739d296e3aa,b89d7997b84b29a5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:42:25.426 INFO 3009027 --- [ocp-schedules-11,6a568739d296e3aa,b89d7997b84b29a5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:42:25.426 INFO 3009027 --- [ocp-schedules-11,6a568739d296e3aa,b89d7997b84b29a5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:42:26.260 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:26.261 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:26.408 INFO 3009027 --- [ocp-schedules-10,7492514a021a98ee,f2ef7641aec7c6e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:42:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:42:26.995 WARN 3009027 --- [ocp-async-4,efbb8858dbab6f93,65c7e0842aa73db3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:26.996 INFO 3009027 --- [ocp-async-7,8b9cfe8f049c89c2,3351c11c0530c887] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,6231232672780830,d42bfa9058d47f0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:28.450 INFO 3009027 --- [ocp-schedules-14,8a473fb3433c2d6b,8e41aa061c1c7d10] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:42:28.451 INFO 3009027 --- [ocp-schedules-14,8a473fb3433c2d6b,8e41aa061c1c7d10] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:42:28.996 WARN 3009027 --- [ocp-async-8,26f3040a9c371862,7333eb29ccef8e19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:28.996 INFO 3009027 --- [ocp-async-3,56b12f5ccb5cb89e,1d2714797856eeea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:29.261 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:29.262 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,43f45bd3fe4924c8,927657f868cd4f24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:30.509 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:42:30.509 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:42:30.512 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:42:30.512 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:42:30.512 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:42:30.512 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:42:30.524 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:42:30.524 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:42:30.524 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:42:30.524 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:42:30.549 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:42:30.554 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:42:30.554 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:42:30.554 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:42:30.554 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:42:30.554 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:42:30.559 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:42:30.851 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:42:30.996 WARN 3009027 --- [ocp-async-10,ed1847f104e96b33,f2a0e3b99bcb89f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:30.996 INFO 3009027 --- [ocp-async-2,453429d18e545d2d,8feef6be1a435531] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:31.190 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.667254302S 2025-04-08 20:42:31.190 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:42:31.246 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:42:31.246 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:42:31.271 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:42:31.313 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:42:31.313 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:42:31.313 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:42:31.313 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:42:31.313 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:42:31.313 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:42:31.320 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:42:31.422 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] o.s.boot.SpringApplication : Started application in 0.909 seconds (JVM running for 1193.039) 2025-04-08 20:42:31.422 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.90997498S 2025-04-08 20:42:31.692 INFO 3009027 --- [ocp-schedules-30,027f49060880188d,e0316782e8fb754e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:42:32.262 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:32.262 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:32.898 INFO 3009027 --- [ocp-schedules-2,c386c31cdcf06a19,82981849d6d1d143] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:42:32.898 INFO 3009027 --- [ocp-schedules-13,c9e87e77c5480419,28f8b635137800e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:42:32.946 INFO 3009027 --- [ocp-async-4,db6fc3649ed3f120,414cf430baf1bc26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:42:32.996 WARN 3009027 --- [ocp-async-7,80774aa250409f7d,d727d619eadd88d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:32.996 INFO 3009027 --- [ocp-async-8,e730b36ddc6e2f3c,9991ffef765d7513] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:33.016 INFO 3009027 --- [ocp-schedules-3,fca9d6bb0f293a52,6d7f5aa7d50380c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:42:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:42:33.177401+08:00, lastProcessTime=2025-04-08T20:42:23.178055+08:00 2025-04-08 20:42:33.292 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:42:33.177401+08:00 2025-04-08 20:42:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,548cb77be9c9065a,9f5100f05392dbe6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:34.996 INFO 3009027 --- [ocp-async-3,b5885d389a79e112,4e423bf8cb8ba56c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:34.996 WARN 3009027 --- [ocp-async-10,70487b1f903dd97f,3fdc8ee574be9f21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:35.262 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:35.263 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,8e554d369ca81e64,48939ce0a2b763e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:36.437 INFO 3009027 --- [ocp-schedules-29,0ed0de12e451c592,2bacd6a1c22a8a26] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:42:36.996 WARN 3009027 --- [ocp-async-1,6ef6bb9352bed8e4,415a8fdea2d6f524] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:36.996 INFO 3009027 --- [ocp-async-9,353a6ac5daa2db55,738cfcab7348e42a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:38.263 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:38.263 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:38.457 INFO 3009027 --- [ocp-schedules-22,913ba418d5afff80,cd356b9dce51e0d0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:42:38.458 INFO 3009027 --- [ocp-schedules-22,913ba418d5afff80,cd356b9dce51e0d0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:42:38.996 WARN 3009027 --- [ocp-async-7,a7f462cb8338fd6a,cae3397f23477007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:38.996 INFO 3009027 --- [ocp-async-4,bb8d5cf3b8494f55,ea6fcd4964f9ba84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,9a813567b323d109,eea9d33d1575538d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:40.002 INFO 3009027 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:42:40.002 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:42:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:42:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:42:40.972 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:42:40.972 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:42:40.973 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:42:40.977 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(2), status=TAKINGOVER 2025-04-08 20:42:40.977 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 20:42:40.996 WARN 3009027 --- [ocp-async-6,309f44e8251daeb6,23df4fd9dd1bf165] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:40.997 INFO 3009027 --- [ocp-async-8,5ba6695f19284162,35c8eacc99eb2293] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:41.263 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:41.264 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,009826d7fcf03cfb,54976ab94c1561c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:42.946 INFO 3009027 --- [ocp-async-3,262754fe56d842e1,14f0af651390a3d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:42:42.997 WARN 3009027 --- [ocp-async-5,8cf54c2c0544ddd3,ef9ff0c5f823db46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:42.997 INFO 3009027 --- [ocp-async-9,9b7e38a1395304d5,add2fec2db98c65d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:43.017 INFO 3009027 --- [ocp-schedules-14,c81fe05767e258c9,ac536f0705244b78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:42:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:42:43.177381+08:00, lastProcessTime=2025-04-08T20:42:33.177401+08:00 2025-04-08 20:42:43.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:42:43.177381+08:00 2025-04-08 20:42:44.264 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:44.265 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:44.997 WARN 3009027 --- [ocp-async-4,4e50c2e343049f9f,daed5e95147ba45f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:44.997 INFO 3009027 --- [ocp-async-9,0bcd7b85d371b625,fae9a083d9572fc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,b7638e6b15b2dbfc,21acf3c258017e79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:45.428 INFO 3009027 --- [ocp-schedules-1,2e0d2f1750cdc9b5,5da8b3679831815e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:42:45.446 INFO 3009027 --- [ocp-schedules-1,2e0d2f1750cdc9b5,5da8b3679831815e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:42:45.446 INFO 3009027 --- [ocp-schedules-1,2e0d2f1750cdc9b5,5da8b3679831815e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:42:46.459 INFO 3009027 --- [ocp-schedules-25,0d66fc54f7817b9c,4a4cc9e064dfa2fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:42:46.997 INFO 3009027 --- [ocp-async-6,d819c900c518b34a,e1c7751ab8858359] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:46.997 WARN 3009027 --- [ocp-async-8,fa6844121d356bd1,7e9081d2e99040a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:47.265 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:47.265 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,f13dad8f5c960acd,f4dbbb888b73603b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:48.461 INFO 3009027 --- [ocp-schedules-10,946f017dd1f968b6,933cd916cb83b793] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:42:48.462 INFO 3009027 --- [ocp-schedules-10,946f017dd1f968b6,933cd916cb83b793] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:42:48.997 WARN 3009027 --- [ocp-async-1,d4b43ecb284498fa,b8f6890f9f8aae01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:48.997 INFO 3009027 --- [ocp-async-5,293eed76845219db,14f20db6f78258c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:50.265 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:50.266 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:50.997 WARN 3009027 --- [ocp-async-5,5e2cae1afb7f557e,c1e5e9e7ecb3e680] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:50.997 INFO 3009027 --- [ocp-async-2,3c15ae2a55c192f5,ffb77c4d6ee64f75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,84a0c22058b1c5e6,3dab3d84359f5ca1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:52.896 INFO 3009027 --- [ocp-schedules-30,475cd97fd4361b12,173abddd8b8368f7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:42:52.896 INFO 3009027 --- [ocp-schedules-8,2af9ccc631a4d038,816404b8f03032ac] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:42:52.897 INFO 3009027 --- [ocp-schedules-8,2af9ccc631a4d038,816404b8f03032ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:52.897 INFO 3009027 --- [ocp-schedules-3,4d7a93c739991ab6,c8da301436f1556e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-3,4d7a93c739991ab6,c8da301436f1556e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:52.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-18,233b52b869c293bc,307c4d94e186e133] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-18,233b52b869c293bc,307c4d94e186e133] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-19,0d2f4d630ef178bc,ac03bc4927641d4c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-19,0d2f4d630ef178bc,ac03bc4927641d4c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-11,533f9c26bff9dac5,c8914888324777c3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:42:52.898 INFO 3009027 --- [ocp-schedules-11,533f9c26bff9dac5,c8914888324777c3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:42:52.909 INFO 3009027 --- [ocp-schedules-4,33df6f1e566aef28,58b765e28b451456] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:42:52.909 INFO 3009027 --- [ocp-schedules-4,e1f22169c6333825,029d333534859d67] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:42:52.909 INFO 3009027 --- [ocp-schedules-4,e1f22169c6333825,029d333534859d67] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:42:52.914 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:42:52.914 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:42:52.915 INFO 3009027 --- [ocp-schedules-15,494474aadaa39f04,f22ae54b854031fb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:42:52.946 INFO 3009027 --- [ocp-async-2,80d25d5c3268790f,25a3a5a72a0109b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:42:52.989 INFO 3009027 --- [ocp-schedules-29,3e521b0e3b0646e1,1bf501680dd2b5f9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:42:52.997 WARN 3009027 --- [ocp-async-8,b1c28ba25c5d51e6,60244b07515ab177] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:52.997 INFO 3009027 --- [ocp-async-6,5e0f922716dc0898,c235e9bedde5e507] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:53.019 INFO 3009027 --- [ocp-schedules-25,acd9ac3e77459388,8c52714a9ddf49c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:42:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:42:53.176339+08:00, lastProcessTime=2025-04-08T20:42:43.177381+08:00 2025-04-08 20:42:53.266 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:53.267 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:53.315 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:42:53.176339+08:00 2025-04-08 20:42:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,12fec395306573eb,0dafcb88642ba1fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:54.998 WARN 3009027 --- [ocp-async-6,b7ccb7fb7314eda1,35556104d32bc8a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:54.998 INFO 3009027 --- [ocp-async-7,eb628ca1b7ff1368,6f817bc114e630bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:56.267 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:56.267 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:42:56.494 INFO 3009027 --- [ocp-schedules-18,5b407287f71c89c2,6e8193f63462b8c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:42:56.998 WARN 3009027 --- [ocp-async-4,b0c459c06d849129,ff0ef4304a12c1e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:56.998 INFO 3009027 --- [ocp-async-5,9c7c2c95520808b5,473ea166ac57d6ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,54190999eeea92f5,0f1e5083760ca9f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116180, interval=60 2025-04-08 20:42:58.509 INFO 3009027 --- [ocp-schedules-22,932d412fff94251e,5dc6846199b6cff7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:42:58.510 INFO 3009027 --- [ocp-schedules-22,932d412fff94251e,5dc6846199b6cff7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:42:58.998 WARN 3009027 --- [ocp-async-5,a1c0bd7a83d5054b,964f9140d3c596e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:42:58.998 INFO 3009027 --- [ocp-async-1,e61708f23696e100,b9a2f95ef9537cab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:42:59.267 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:42:59.268 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,a748bc7becb7a934,fa9878ea2022ce7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:00.998 WARN 3009027 --- [ocp-async-6,12cfcb4ef94dfd41,d3f301cebb6bf30c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:00.998 INFO 3009027 --- [ocp-async-3,3fee9f3b2c46f433,770a2440d8380481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:02.268 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:02.269 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:02.898 INFO 3009027 --- [ocp-schedules-8,2af9ccc631a4d038,816404b8f03032ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:43:02.898 INFO 3009027 --- [ocp-schedules-3,4d7a93c739991ab6,c8da301436f1556e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:43:02.946 INFO 3009027 --- [ocp-async-3,b442f80ddafdf2fb,957964a1d538020b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:43:02.946 INFO 3009027 --- [ocp-schedules-21,8c3d65eeb48b2bf1,e1004c915d4fd95e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:43:02.947 INFO 3009027 --- [ocp-schedules-21,8c3d65eeb48b2bf1,e1004c915d4fd95e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:02.947 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:43:02.998 INFO 3009027 --- [ocp-async-4,c9ac944f10c6c4e9,d2a1f4c2d086455e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:02.998 WARN 3009027 --- [ocp-async-9,6d165eacad30b935,b7ba6b1264efdf95] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:03.020 INFO 3009027 --- [ocp-schedules-5,9cdd9b4c8448f254,f39e947dcd545f88] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:43:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:43:03.176389+08:00, lastProcessTime=2025-04-08T20:42:53.176339+08:00 2025-04-08 20:43:03.289 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:43:03.176389+08:00 2025-04-08 20:43:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,08d42f3dd6a51f66,cf45e1a59afd5275] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:04.998 WARN 3009027 --- [ocp-async-4,44bd74c9ccdd252c,3c063af101060dfc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:04.998 INFO 3009027 --- [ocp-async-3,054fa78f323eac94,3aca6b428476b4cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:05.269 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:05.270 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:05.447 INFO 3009027 --- [ocp-schedules-20,b29fcd6b6b7d2328,d9b779215dd4d22e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:43:05.469 INFO 3009027 --- [ocp-schedules-20,b29fcd6b6b7d2328,d9b779215dd4d22e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:43:05.469 INFO 3009027 --- [ocp-schedules-20,b29fcd6b6b7d2328,d9b779215dd4d22e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:43:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,df535dece8820ef4,c19b92dfebb3a1a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:06.515 INFO 3009027 --- [ocp-schedules-13,2ecfa76c48831194,5893c252680c50bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:43:06.999 WARN 3009027 --- [ocp-async-10,29985c34f0872c7d,763d9e6a60ef9d10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:06.999 INFO 3009027 --- [ocp-async-2,f244ff9b974699b8,6bdbd3a31a2f5251] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:08.270 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:08.270 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:08.577 INFO 3009027 --- [ocp-schedules-10,9baa5ebad6a78b8d,b71cb425262bf9ff] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:43:08.578 INFO 3009027 --- [ocp-schedules-10,9baa5ebad6a78b8d,b71cb425262bf9ff] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:43:08.999 WARN 3009027 --- [ocp-async-2,741b9b0fe354fc06,3a646b3a305f42fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:08.999 INFO 3009027 --- [ocp-async-5,e988cda9e1ac1aac,95205b03b0058f0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,5a824fc438c32478,b965b6e24fbfedc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:10.001 INFO 3009027 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:43:10.001 INFO 3009027 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:43:10.994 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:43:10.999 WARN 3009027 --- [ocp-async-6,82f7570ff142c2ca,8f1a1d1036d2127a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:10.999 INFO 3009027 --- [ocp-async-4,5de70003aeccab62,b6f87a0f3f4dadc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:11.270 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:11.271 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,63c139e472336f51,5728f1e14e7c1a8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:12.408 INFO 3009027 --- [prometheus-cache-monitor1,9d2c757b14c99440,7bdbded99af57418] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1680,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":422,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:43:12.408 INFO 3009027 --- [prometheus-cache-monitor1,9d2c757b14c99440,7bdbded99af57418] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:43:12.946 INFO 3009027 --- [ocp-async-4,28a30db3ad1d9743,bbb939777a01fe28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:43:12.948 INFO 3009027 --- [ocp-schedules-21,8c3d65eeb48b2bf1,e1004c915d4fd95e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:43:12.999 WARN 3009027 --- [ocp-async-3,c81e31b7da07bac9,2fa923695052db05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:12.999 INFO 3009027 --- [ocp-async-9,dd250d8b5def47f4,057dcb69717b41ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:13.022 INFO 3009027 --- [ocp-schedules-6,5506c0b210c372ae,84c90aa0e1cd7a34] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:43:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:43:13.177116+08:00, lastProcessTime=2025-04-08T20:43:03.176389+08:00 2025-04-08 20:43:13.241 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:39:13.046, end=2025-04-08T20:41:13.233} 2025-04-08 20:43:13.247 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:43:13.247 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:43:13.263 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:43:13.177116+08:00 2025-04-08 20:43:14.271 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:14.272 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:14.999 WARN 3009027 --- [ocp-async-8,b8355c96e969d94a,af1b487d10eb4380] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:14.999 INFO 3009027 --- [ocp-async-7,5b8da4aa5b57b3de,2ed1bbea618c7062] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,8821731543dd4a48,d115a75153d86c59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:15.664 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116120, nodeCount=2, affectRows=2 2025-04-08 20:43:15.664 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116120, nodeCount=1, validCount=1 2025-04-08 20:43:15.665 INFO 3009027 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116120, nodeCount=420, affectRows=420 2025-04-08 20:43:15.668 INFO 3009027 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116120, nodeCount=418, validCount=418 2025-04-08 20:43:16.541 INFO 3009027 --- [ocp-schedules-22,8981a991fce766f1,88963ce7fc3f8fa8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:43:16.999 WARN 3009027 --- [ocp-async-1,db170fb879390262,1f0250d8d0f6e653] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:16.999 INFO 3009027 --- [ocp-async-7,785459264845c9be,994877c67f7595d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:17.272 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:17.272 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,f71544b46d2d936c,d236a46a0a44e311] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:18.613 INFO 3009027 --- [ocp-schedules-15,5d407192b8392123,998439fa56727814] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:43:18.614 INFO 3009027 --- [ocp-schedules-15,5d407192b8392123,998439fa56727814] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:43:18.999 WARN 3009027 --- [ocp-async-10,ba0109989b4dd6b6,df581eca1304eab2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:19.000 INFO 3009027 --- [ocp-async-4,6a8cb244955574c5,b627beea46032634] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:20.272 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:20.273 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:21.000 INFO 3009027 --- [ocp-async-8,9a018777ccbe6000,ac87742deef19efb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:21.000 WARN 3009027 --- [ocp-async-6,438544c747d9c182,c2e93390d2b3fb6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,286bc2c2661ab002,b8e1402d77149098] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:22.895 INFO 3009027 --- [ocp-schedules-16,d5ff75d1ad977003,f9a99ff07dec6965] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:43:22.896 INFO 3009027 --- [ocp-schedules-22,d067a09530d99c92,8421f13afe5e924f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:43:22.896 INFO 3009027 --- [ocp-schedules-13,b03fff35af198943,22d54ce77b9856ea] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:43:22.896 INFO 3009027 --- [ocp-schedules-14,b1f7c82a39e6794c,ea4ac1f6e2c5b732] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:43:22.896 INFO 3009027 --- [ocp-schedules-25,f848b941bd44fc38,d5c773e0bc35313a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:43:22.897 INFO 3009027 --- [ocp-schedules-13,b03fff35af198943,22d54ce77b9856ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:22.897 INFO 3009027 --- [ocp-schedules-25,f848b941bd44fc38,d5c773e0bc35313a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:43:22.898 INFO 3009027 --- [ocp-schedules-8,42cc2affa68d60a8,a87fd3b986d626c5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:43:22.898 INFO 3009027 --- [ocp-schedules-5,a4a4faaf4ca50df5,6ef5d9e6b3e31ced] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:43:22.898 INFO 3009027 --- [ocp-schedules-8,42cc2affa68d60a8,a87fd3b986d626c5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:43:22.898 INFO 3009027 --- [ocp-schedules-5,a4a4faaf4ca50df5,6ef5d9e6b3e31ced] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:43:22.898 INFO 3009027 --- [ocp-schedules-10,b306e09886732d56,e67312447889bd7d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:43:22.898 INFO 3009027 --- [ocp-schedules-10,b306e09886732d56,e67312447889bd7d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:43:22.903 INFO 3009027 --- [ocp-schedules-14,b1f7c82a39e6794c,ea4ac1f6e2c5b732] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:43:22.915 INFO 3009027 --- [ocp-schedules-4,e1f22169c6333825,029d333534859d67] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:43:22.915 INFO 3009027 --- [ocp-schedules-4,ce90b853d6321b84,435ef4c99378433f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:43:22.916 INFO 3009027 --- [ocp-schedules-4,ce90b853d6321b84,435ef4c99378433f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:22.919 INFO 3009027 --- [ocp-schedules-21,8b43b1baba1bc76f,589b386ad5d6604c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:43:22.920 INFO 3009027 --- [ocp-schedules-21,8b43b1baba1bc76f,589b386ad5d6604c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:43:22.920 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:43:22.920 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:43:22.946 INFO 3009027 --- [ocp-async-1,78a42295b939ebdd,0c07c9ae0f0bdf39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:43:22.965 INFO 3009027 --- [ocp-schedules-16,d5ff75d1ad977003,f9a99ff07dec6965] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:43:22.965 INFO 3009027 --- [ocp-schedules-27,a78f98a9bd680d32,b4ba19d63b3b2b5f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:43:22.976 INFO 3009027 --- [ocp-schedules-11,5fe63c0be151459c,cb797a8b731cc9df] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:43:22.976+08:00, clusters=[2] 2025-04-08 20:43:22.977 INFO 3009027 --- [ocp-schedules-18,1ff70952ae3d22fa,d4ccc1677832fcae] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:43:22.977 INFO 3009027 --- [ocp-schedules-11,5fe63c0be151459c,cb797a8b731cc9df] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:43:22.990 INFO 3009027 --- [ocp-schedules-28,e4d455872b37342b,f6d4d7b1e567cf61] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:43:22.998 INFO 3009027 --- [ocp-schedules-20,27015550fe0425f5,33a4bab83d71ffe7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:43:22.998 INFO 3009027 --- [ocp-schedules-20,27015550fe0425f5,33a4bab83d71ffe7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:43:23.000 INFO 3009027 --- [ocp-async-10,167041075aa37acd,6c06f994ff055b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:23.000 WARN 3009027 --- [ocp-async-3,8c2f8f45a2d068dc,0ecb556c051852a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:23.000 INFO 3009027 --- [ocp-schedules-15,9358e611b822f9de,5025e0f67fb6e0ce] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:43:23+08:00, clusters=[2] 2025-04-08 20:43:23.002 INFO 3009027 --- [ocp-schedules-15,9358e611b822f9de,5025e0f67fb6e0ce] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:43:23.015 INFO 3009027 --- [ocp-schedules-10,65dadf483b659b5a,a27a826139c75ff2] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:43:23.015 INFO 3009027 --- [ocp-schedules-10,65dadf483b659b5a,a27a826139c75ff2] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:43:23.016 INFO 3009027 --- [ocp-schedules-14,70c2437e50956d68,1da310429ca61028] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:43:23.018 INFO 3009027 --- [ocp-schedules-14,70c2437e50956d68,1da310429ca61028] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:43:23.018 INFO 3009027 --- [ocp-schedules-14,70c2437e50956d68,1da310429ca61028] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:43:23.019 INFO 3009027 --- [ocp-schedules-3,9b54a3e09b066640,e6be652209c63ffe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:43:23.019 INFO 3009027 --- [ocp-schedules-3,9b54a3e09b066640,e6be652209c63ffe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:43:23.023 INFO 3009027 --- [ocp-schedules-26,3736f32cd2487aa8,f1c70d775f2da4da] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:43:23.023 INFO 3009027 --- [ocp-schedules-3,9b54a3e09b066640,e6be652209c63ffe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:43:23.023 INFO 3009027 --- [ocp-schedules-3,9b54a3e09b066640,e6be652209c63ffe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:43:23.068 INFO 3009027 --- [ocp-schedules-23,da42ee687e728555,81d32fb2e2983ac0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:43:23.110 INFO 3009027 --- [ocp-schedules-17,fd3ea76d9b45237d,a23d3a661960dd46] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:43:23.119 INFO 3009027 --- [ocp-schedules-6,eab78cc1a6b1c200,5820933c9a00e91f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:43:23.119 INFO 3009027 --- [ocp-schedules-6,eab78cc1a6b1c200,5820933c9a00e91f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:43:23.122 INFO 3009027 --- [ocp-schedules-6,eab78cc1a6b1c200,5820933c9a00e91f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:43:23.122 INFO 3009027 --- [ocp-schedules-6,eab78cc1a6b1c200,5820933c9a00e91f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:43:23.125 INFO 3009027 --- [ocp-schedules-6,eab78cc1a6b1c200,5820933c9a00e91f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:43:23.125 INFO 3009027 --- [ocp-schedules-6,eab78cc1a6b1c200,5820933c9a00e91f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:43:23.143 INFO 3009027 --- [ocp-schedules-2,5cc756979166f3f4,c20aa34c9709d28d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:43:23.144 INFO 3009027 --- [ocp-schedules-2,5cc756979166f3f4,c20aa34c9709d28d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:23.159 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:43:23.160 ERROR 3009027 --- [ocp-schedules-2,5cc756979166f3f4,c20aa34c9709d28d] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:43:23.160 INFO 3009027 --- [ocp-schedules-2,5cc756979166f3f4,c20aa34c9709d28d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:43:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:43:23.176130+08:00, lastProcessTime=2025-04-08T20:43:13.177116+08:00 2025-04-08 20:43:23.273 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:23.274 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:23.285 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:43:23.176130+08:00 2025-04-08 20:43:23.934 INFO 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:43:23.934 INFO 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:43:23.936 INFO 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:43:23.937 WARN 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:43:23.938 INFO 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:43:23.939 WARN 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:43:23.942 INFO 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:43:23.943 WARN 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:43:23.944 ERROR 3009027 --- [ocp-schedules-20,880c8ef3c257fb92,cd212e5b40de3a9f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:43:24.169 INFO 3009027 --- [ocp-schedules-26,fca3ffac7288628e,4fc374722d3facdf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:43:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,2653f7b74442506e,ffd28cac68aac6a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:25.000 WARN 3009027 --- [ocp-async-2,72ea75ba3b1d3828,f72c8461f59ed096] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:25.000 INFO 3009027 --- [ocp-async-10,9e10897fe3516e8e,113fba2fe0ad4874] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:25.470 INFO 3009027 --- [ocp-schedules-26,362684d8bfb695b0,543ecf5d4adcc253] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:43:25.485 INFO 3009027 --- [ocp-schedules-26,362684d8bfb695b0,543ecf5d4adcc253] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:43:25.485 INFO 3009027 --- [ocp-schedules-26,362684d8bfb695b0,543ecf5d4adcc253] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:43:26.274 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:26.275 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:26.561 INFO 3009027 --- [ocp-schedules-7,671f99a1a7029cda,0e605d93fac1c4f0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:43:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:43:27.000 WARN 3009027 --- [ocp-async-5,8dd315ecd33f476b,4298f1e5097f4465] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:27.000 INFO 3009027 --- [ocp-async-10,1e3c595c4af4d99c,113f84cb0bd1316f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,c0ee3f2bc621eba6,0f96e37a680c7ed5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:28.617 INFO 3009027 --- [ocp-schedules-28,f3a12b6bb0983a91,bc5cf42b23d1522e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:43:28.618 INFO 3009027 --- [ocp-schedules-28,f3a12b6bb0983a91,bc5cf42b23d1522e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:43:29.000 WARN 3009027 --- [ocp-async-3,67ff984b750150d8,d4d86d86a7860710] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:29.000 INFO 3009027 --- [ocp-async-1,3113a6d3cd84ef43,114247c7e8b48e89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:29.275 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:29.275 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,5b20cc932bbb472e,27cbfde99d332455] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:31.000 WARN 3009027 --- [ocp-async-1,cc1dec2a81776fa2,1ec6c5b077472fcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:31.000 INFO 3009027 --- [ocp-async-8,56e1fd1528b90b9e,449a9ff8d35feebd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:32.276 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:32.276 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:32.897 INFO 3009027 --- [ocp-schedules-13,b03fff35af198943,22d54ce77b9856ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:43:32.898 INFO 3009027 --- [ocp-schedules-25,f848b941bd44fc38,d5c773e0bc35313a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:43:32.947 INFO 3009027 --- [ocp-async-6,a1ccce1721a71bc2,0bb3374c460b93c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:43:33.001 WARN 3009027 --- [ocp-async-10,45f42a6823f8b9ee,fffe92134f9f4f12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:33.001 INFO 3009027 --- [ocp-async-3,b244e051522325a3,c15e416b368744f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:33.025 INFO 3009027 --- [ocp-schedules-27,69d160df7dcf922b,093972312f105847] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:43:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:43:33.177568+08:00, lastProcessTime=2025-04-08T20:43:23.176130+08:00 2025-04-08 20:43:33.285 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:43:33.177568+08:00 2025-04-08 20:43:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,708a0380c9feac18,0cd917f726622076] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:35.001 WARN 3009027 --- [ocp-async-7,5cd5285e9b0b3397,4d88f39860586336] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:35.001 INFO 3009027 --- [ocp-async-1,d979368dc55dfe29,d5e34c2eb165084a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:35.276 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:35.277 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,1e6c10d173d693b6,dc7b2b79bb15d206] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:36.591 INFO 3009027 --- [ocp-schedules-19,1bd22087db1fee74,b9b7efb893e7ad92] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:43:37.001 WARN 3009027 --- [ocp-async-2,1eaf1475c13afeb7,4cd89badaa455227] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:37.001 INFO 3009027 --- [ocp-async-8,2c234911dc5ae4f2,5dc468dc9dac0af9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:38.277 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:38.278 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:38.623 INFO 3009027 --- [ocp-schedules-7,e6e2991d2a1106ca,0b6f4eafa809bde9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:43:38.624 INFO 3009027 --- [ocp-schedules-7,e6e2991d2a1106ca,0b6f4eafa809bde9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:43:39.001 WARN 3009027 --- [ocp-async-8,47f483219950fad9,83934713f8b28a8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:39.001 INFO 3009027 --- [ocp-async-6,8666e21eb09b9d93,58db5247c54a0880] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,c2d45461c1bbf1f7,04c6a6df628372d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:40.002 INFO 3009027 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:43:40.002 INFO 3009027 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:43:41.001 WARN 3009027 --- [ocp-async-7,fc6bfe34cc737052,e95a27e4cfa03de1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:41.001 INFO 3009027 --- [ocp-async-9,c1b8e89db0239243,060fb8eb3338b191] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:41.278 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:41.278 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,2991651d47513dc8,14d6f931a21b9a19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:42.947 INFO 3009027 --- [ocp-async-9,7e22cbf8ccedcd26,bb1ae1b46738c445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:43:43.001 WARN 3009027 --- [ocp-async-2,76c15df7d0eefcd0,1f07726c77adaf19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:43.001 INFO 3009027 --- [ocp-async-2,30f373508b90d5dc,c0e54b9f86717dee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:43.026 INFO 3009027 --- [ocp-schedules-14,cbdab54dcad3b977,546e8f243a68aa3a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:43:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:43:43.175981+08:00, lastProcessTime=2025-04-08T20:43:33.177568+08:00 2025-04-08 20:43:43.283 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:43:43.175981+08:00 2025-04-08 20:43:44.278 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:44.279 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:45.001 WARN 3009027 --- [ocp-async-8,ffd8ba284fd77bde,490934f81db29700] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:45.002 INFO 3009027 --- [ocp-async-4,8280cfef537b8f71,b2481aa7a07b4591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,1b3431ad083351c2,bf975a74ff6fb29f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:45.486 INFO 3009027 --- [ocp-schedules-8,1797d8f603a4b157,b53f8ff434ef1feb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:43:45.534 INFO 3009027 --- [ocp-schedules-8,1797d8f603a4b157,b53f8ff434ef1feb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:43:45.534 INFO 3009027 --- [ocp-schedules-8,1797d8f603a4b157,b53f8ff434ef1feb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:43:46.611 INFO 3009027 --- [ocp-schedules-19,4b53cd9ade684ace,e97d879a6c51bc75] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:43:47.002 WARN 3009027 --- [ocp-async-7,fa221fffd53d0e2b,4484691b073ad032] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:47.002 INFO 3009027 --- [ocp-async-3,133f915e4a4ce8cd,1b4bd8593c5f205b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:47.279 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:47.280 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,ba1b284279b1b1d5,75da2d0651e9504b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:48.629 INFO 3009027 --- [ocp-schedules-3,559dafb19974ca21,37789d3b4539ea0c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:43:48.630 INFO 3009027 --- [ocp-schedules-3,559dafb19974ca21,37789d3b4539ea0c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:43:49.002 WARN 3009027 --- [ocp-async-3,a3dccebd15d06892,c21b19691784f331] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:49.002 INFO 3009027 --- [ocp-async-10,30401e21840e6d8a,c0a83f4065f6328f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:50.280 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:50.281 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:51.002 INFO 3009027 --- [ocp-async-2,fae2309264cd8c83,c508a452a8b7c291] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:51.002 WARN 3009027 --- [ocp-async-6,457dce6e78e29052,eb744c48d6101a9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,3327af2bd328fc54,b1d44525445a4844] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:52.896 INFO 3009027 --- [ocp-schedules-3,b48fc639bb5dd855,708c7d192fe30722] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:43:52.896 INFO 3009027 --- [ocp-schedules-17,7332b5e97557e118,2e7fbb17d344e0b6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:43:52.896 INFO 3009027 --- [ocp-schedules-12,483167f08e200480,1395746549afff82] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:43:52.897 INFO 3009027 --- [ocp-schedules-17,7332b5e97557e118,2e7fbb17d344e0b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:52.897 INFO 3009027 --- [ocp-schedules-12,483167f08e200480,1395746549afff82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:43:52.899 INFO 3009027 --- [ocp-schedules-1,d90b3292e3c74887,66d0439972493ed3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:43:52.899 INFO 3009027 --- [ocp-schedules-5,71e6cc83d8d03195,a5c9a59c14c7c68b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:43:52.899 INFO 3009027 --- [ocp-schedules-8,a3260393c1ce6429,27b6486034f29303] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:43:52.899 INFO 3009027 --- [ocp-schedules-1,d90b3292e3c74887,66d0439972493ed3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:43:52.899 INFO 3009027 --- [ocp-schedules-8,a3260393c1ce6429,27b6486034f29303] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:43:52.899 INFO 3009027 --- [ocp-schedules-5,71e6cc83d8d03195,a5c9a59c14c7c68b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:43:52.902 INFO 3009027 --- [ocp-schedules-20,1550e91713604f5a,89564edc4594cda2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:43:52.921 INFO 3009027 --- [ocp-schedules-4,ce90b853d6321b84,435ef4c99378433f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:43:52.921 INFO 3009027 --- [ocp-schedules-4,75833e9fd00afe25,4c03e8d603bfaa03] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:43:52.922 INFO 3009027 --- [ocp-schedules-4,75833e9fd00afe25,4c03e8d603bfaa03] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:43:52.927 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:43:52.927 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:43:52.947 INFO 3009027 --- [ocp-async-2,21db7c7bd9ec6d9d,fc55318bbac71623] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:43:52.992 INFO 3009027 --- [ocp-schedules-29,a9c8973aebd61e91,1433d8e8e1266389] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:43:53.002 WARN 3009027 --- [ocp-async-7,ebc635d8c47fcd12,9bc44ab945fd9bd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:53.002 INFO 3009027 --- [ocp-async-4,da458688e08af39e,4f131522ba9e8810] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:53.028 INFO 3009027 --- [ocp-schedules-7,b10353d4b8649d19,1beaa34572d34ed7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:43:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:43:53.176995+08:00, lastProcessTime=2025-04-08T20:43:43.175981+08:00 2025-04-08 20:43:53.281 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:53.282 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:53.343 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:43:53.176995+08:00 2025-04-08 20:43:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,4ede8842f2b7090a,d244e1b591a262f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:55.002 WARN 3009027 --- [ocp-async-3,4b6547cae911abcd,ff3344270d9f4e3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:55.002 INFO 3009027 --- [ocp-async-4,7d5417a314f7d74f,1e3d127db5dbbc7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:56.282 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:56.282 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:43:56.635 INFO 3009027 --- [ocp-schedules-30,4aa8d46e15960ab7,196ded9842269945] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:43:57.002 WARN 3009027 --- [ocp-async-8,65fb986be0de8120,b103ee8dc906f375] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:57.002 INFO 3009027 --- [ocp-async-6,895a86a37b2a5efa,d92c8c08b4f0c62e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,51fc651c141b9fb0,665e88a2454eaec6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116240, interval=60 2025-04-08 20:43:58.633 INFO 3009027 --- [ocp-schedules-10,cf4a165e41b4e96f,9376a2536e11da82] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:43:58.634 INFO 3009027 --- [ocp-schedules-10,cf4a165e41b4e96f,9376a2536e11da82] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:43:59.003 WARN 3009027 --- [ocp-async-6,a3103427cc596e74,0f1c958df8a4a76c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:43:59.003 INFO 3009027 --- [ocp-async-7,c5c8a9b031749416,ad599065675df766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:43:59.282 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:43:59.283 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,419dc5c66958d391,ebfbb8000a8b21ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:01.003 WARN 3009027 --- [ocp-async-3,bdf374b1555ce1b4,7cf1f818b0e417fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:01.003 INFO 3009027 --- [ocp-async-1,b6c72287f98cc6a3,a783470ad6f19087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:02.283 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:02.284 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:02.897 INFO 3009027 --- [ocp-schedules-17,7332b5e97557e118,2e7fbb17d344e0b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:44:02.898 INFO 3009027 --- [ocp-schedules-12,483167f08e200480,1395746549afff82] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:44:02.947 INFO 3009027 --- [ocp-async-1,c07ad617bf1bcdcf,1971f98073843263] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:44:03.003 WARN 3009027 --- [ocp-async-8,c430e82d10904727,a3e7f65bfbcd5887] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:03.003 INFO 3009027 --- [ocp-async-2,b825894c81804ec5,fddc8b40425f6214] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:03.029 INFO 3009027 --- [ocp-schedules-14,9726f4f7eca94079,07e893861da7cd09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:44:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:44:03.177143+08:00, lastProcessTime=2025-04-08T20:43:53.176995+08:00 2025-04-08 20:44:03.262 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:44:03.177143+08:00 2025-04-08 20:44:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,cd847ef613121b88,08d18c17fe82664f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:05.003 WARN 3009027 --- [ocp-async-6,9c8f060040bdec53,03b3b7f83659dfe0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:05.003 INFO 3009027 --- [ocp-async-7,dd4bd3658ff17833,4dc296e55673e595] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:05.284 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:05.285 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:05.536 INFO 3009027 --- [ocp-schedules-7,8665f724484be8f7,5851bd517d561588] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:44:05.541 INFO 3009027 --- [ocp-schedules-7,8665f724484be8f7,5851bd517d561588] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:44:05.541 INFO 3009027 --- [ocp-schedules-7,8665f724484be8f7,5851bd517d561588] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:44:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,afd6bf702b3b1b85,3d532a39959b8a68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:06.657 INFO 3009027 --- [ocp-schedules-13,daa813fa00df5b14,50856049fb28ba54] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:44:07.003 WARN 3009027 --- [ocp-async-9,436f1466411711ee,c13d59410407cbe2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:07.003 INFO 3009027 --- [ocp-async-10,e92f16da3c0afe16,55f4e54a496ef3cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:08.285 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:08.285 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:08.637 INFO 3009027 --- [ocp-schedules-24,59ad57edc4f8bfa7,cb106e4981a5bfa2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:44:08.638 INFO 3009027 --- [ocp-schedules-24,59ad57edc4f8bfa7,cb106e4981a5bfa2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:44:09.003 WARN 3009027 --- [ocp-async-1,940f9eaad2fa980c,f510c0ebc3237a2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:09.003 INFO 3009027 --- [ocp-async-10,ec21d7d227fd1cac,ea4e5b56d4c7b560] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,0a7000ca3958022c,6c70198203e1f194] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:10.001 INFO 3009027 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:44:10.001 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:44:10.989 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:44:11.004 WARN 3009027 --- [ocp-async-5,8dee6c8610b8c31e,12336a850bdd9a2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:11.004 INFO 3009027 --- [ocp-async-6,42eece2b19615bbf,7230eed32f75eb25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:11.286 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:11.286 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,d463f233541047fb,c66bfd669fbbef64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:12.408 INFO 3009027 --- [prometheus-cache-monitor1,e094b1b3230b1cfc,e21b4177a65adf8c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1684,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":422,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:44:12.408 INFO 3009027 --- [prometheus-cache-monitor1,e094b1b3230b1cfc,e21b4177a65adf8c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:44:12.947 INFO 3009027 --- [ocp-async-7,aa9df54f457ea638,21d5ccb58e7d8fee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:44:12.948 INFO 3009027 --- [ocp-schedules-25,31153f01a6ba4d7f,28cd6e9273dbf54d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:44:12.949 INFO 3009027 --- [ocp-schedules-25,31153f01a6ba4d7f,28cd6e9273dbf54d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:12.949 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:44:13.004 INFO 3009027 --- [ocp-async-9,5534c89a1e77be45,3b0b50fe8c119a4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:13.004 WARN 3009027 --- [ocp-async-3,0c92abc3085836ad,09de843fda2bfb28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:13.030 INFO 3009027 --- [ocp-schedules-26,7b2b492e981efc20,65c7853bc20970d6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:44:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:44:13.176678+08:00, lastProcessTime=2025-04-08T20:44:03.177143+08:00 2025-04-08 20:44:13.259 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:44:13.176678+08:00 2025-04-08 20:44:14.286 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:14.287 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:15.004 WARN 3009027 --- [ocp-async-10,0c44c5cc182dd3ac,27790298a060cbf4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:15.004 INFO 3009027 --- [ocp-async-4,432e5107e3526c48,56897699d61fd7d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,c20f3c8bcaf895c3,6b745e8daa19507c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:15.693 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116180, nodeCount=422, affectRows=422 2025-04-08 20:44:15.696 INFO 3009027 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116180, nodeCount=419, validCount=419 2025-04-08 20:44:16.685 INFO 3009027 --- [ocp-schedules-29,1da8712dd815d965,1ce5ba4db7f4fd0c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:44:17.004 WARN 3009027 --- [ocp-async-2,a27e71c9a76c5295,c0e30bd68ddb5972] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:17.004 INFO 3009027 --- [ocp-async-6,a8642a650f3cf478,0c82dba4aa02f948] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:17.287 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:17.288 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,5f5343c2f0794859,02c7271d7bbbee90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:18.683 INFO 3009027 --- [ocp-schedules-28,43beb1a0c10f123e,9c4dc50acc32006d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:44:18.684 INFO 3009027 --- [ocp-schedules-28,43beb1a0c10f123e,9c4dc50acc32006d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:44:19.004 WARN 3009027 --- [ocp-async-3,b4ede7215e011d4e,22e13c80368a4aa9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:19.004 INFO 3009027 --- [ocp-async-8,5fb4bd858a6d8234,9805146d30ab7977] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:20.288 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:20.288 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:21.004 WARN 3009027 --- [ocp-async-1,cd3fa1794e91a2f7,66c78d08b911c1e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:21.004 INFO 3009027 --- [ocp-async-9,bf18202b7150ccc0,3f56a125878ff138] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,07644312fae43a7b,8a398c3470957404] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:22.895 INFO 3009027 --- [ocp-schedules-8,49da65b5332dcf50,06e9e0724ccba9bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:44:22.896 INFO 3009027 --- [ocp-schedules-15,c90ac7dbeb50334a,5d6d0be801ea0256] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:44:22.896 INFO 3009027 --- [ocp-schedules-19,187d96ff9881359f,503831f0ff6bc9c2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:44:22.896 INFO 3009027 --- [ocp-schedules-14,b15a1c9c6ef1f624,ec1b9e94ca4bfcdf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:44:22.896 INFO 3009027 --- [ocp-schedules-1,ff44d96ca92ec8e2,312d2dc1cc9a83e5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:44:22.897 INFO 3009027 --- [ocp-schedules-19,187d96ff9881359f,503831f0ff6bc9c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:22.897 INFO 3009027 --- [ocp-schedules-1,ff44d96ca92ec8e2,312d2dc1cc9a83e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:44:22.898 INFO 3009027 --- [ocp-schedules-30,c1aa0e268a9ba492,fec52262ecd02886] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:44:22.898 INFO 3009027 --- [ocp-schedules-30,c1aa0e268a9ba492,fec52262ecd02886] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:44:22.898 INFO 3009027 --- [ocp-schedules-26,9b5620e66c544def,219fd866e41cea97] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:44:22.899 INFO 3009027 --- [ocp-schedules-26,9b5620e66c544def,219fd866e41cea97] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:44:22.899 INFO 3009027 --- [ocp-schedules-20,2258726a23712adc,b27cf4cbd03fabe5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:44:22.899 INFO 3009027 --- [ocp-schedules-20,2258726a23712adc,b27cf4cbd03fabe5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:44:22.903 INFO 3009027 --- [ocp-schedules-27,14c9e833885d6913,bd3e4b9ed5dcec38] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 20:44:22.903 INFO 3009027 --- [ocp-schedules-27,14c9e833885d6913,bd3e4b9ed5dcec38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 20:44:22.904 INFO 3009027 --- [ocp-schedules-14,b15a1c9c6ef1f624,ec1b9e94ca4bfcdf] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:44:22.914 INFO 3009027 --- [pool-overview-container-1,bb7ab7fb48e128b5,74297efa39415488] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:44:22.914 INFO 3009027 --- [pool-overview-container-5,9d8b5f7a34a13206,e67db8ae86a37ddf] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:44:22.914 INFO 3009027 --- [ocp-schedules-5,e04706aea7f446c8,aa5541d2f86801f3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":420,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:44:22.914 INFO 3009027 --- [ocp-schedules-5,e04706aea7f446c8,aa5541d2f86801f3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:44:22.918 INFO 3009027 --- [ocp-schedules-8,49da65b5332dcf50,06e9e0724ccba9bc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:44:22.919 INFO 3009027 --- [pool-11-thread-4,9d8b5f7a34a13206,077907ee53778c1b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 20:44:22.919 INFO 3009027 --- [pool-11-thread-4,9d8b5f7a34a13206,077907ee53778c1b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 20:44:22.919 INFO 3009027 --- [pool-11-thread-1,9d8b5f7a34a13206,a837587ddec6f509] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 20:44:22.919 INFO 3009027 --- [pool-11-thread-2,9d8b5f7a34a13206,05f4ce908a411c9c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 20:44:22.919 INFO 3009027 --- [pool-11-thread-3,9d8b5f7a34a13206,f1ea79667a083d08] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 20:44:22.922 INFO 3009027 --- [ForkJoinPool-7-worker-4,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:44:22.922 INFO 3009027 --- [ForkJoinPool-7-worker-4,9d8b5f7a34a13206,6e25c523256e1414] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 20:44:22.922 INFO 3009027 --- [pool-11-thread-1,9d8b5f7a34a13206,a837587ddec6f509] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 20:44:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-13,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:44:22.923 INFO 3009027 --- [pool-11-thread-4,9d8b5f7a34a13206,077907ee53778c1b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 20:44:22.923 INFO 3009027 --- [pool-11-thread-2,9d8b5f7a34a13206,05f4ce908a411c9c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 20:44:22.923 INFO 3009027 --- [pool-11-thread-2,9d8b5f7a34a13206,05f4ce908a411c9c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 20:44:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-13,9d8b5f7a34a13206,bde757eabfbbd243] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 20:44:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-13,9d8b5f7a34a13206,bde757eabfbbd243] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 20:44:22.923 INFO 3009027 --- [ForkJoinPool-7-worker-13,9d8b5f7a34a13206,bde757eabfbbd243] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 20:44:22.923 INFO 3009027 --- [pool-11-thread-4,9d8b5f7a34a13206,077907ee53778c1b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 20:44:22.923 INFO 3009027 --- [pool-11-thread-3,9d8b5f7a34a13206,f1ea79667a083d08] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 20:44:22.927 INFO 3009027 --- [ocp-schedules-16,eecfe8330d9fb94b,545b7269c62311ee] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:44:22.927 INFO 3009027 --- [ocp-schedules-28,0a659a0c207f40f7,26b8614a0306ef13] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:44:22.927 INFO 3009027 --- [ocp-schedules-4,75833e9fd00afe25,4c03e8d603bfaa03] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:44:22.927 INFO 3009027 --- [ocp-schedules-4,e85f13f94b098d70,053045da5cfacfc3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:44:22.927 INFO 3009027 --- [ocp-schedules-16,eecfe8330d9fb94b,545b7269c62311ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:22.927 INFO 3009027 --- [ocp-schedules-4,e85f13f94b098d70,053045da5cfacfc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:22.932 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:44:22.932 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:44:22.948 INFO 3009027 --- [ocp-async-9,a49b1919803f2421,f19081b947848f8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:44:22.949 INFO 3009027 --- [ocp-schedules-25,31153f01a6ba4d7f,28cd6e9273dbf54d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:44:22.955 INFO 3009027 --- [ocp-schedules-10,368a8958d150df15,bee5780593f27a9b] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:44:22.958 INFO 3009027 --- [ocp-schedules-26,4526a36a1affdf35,dc1f5acf0db643fa] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 20:44:22.959 INFO 3009027 --- [ocp-schedules-26,4526a36a1affdf35,dc1f5acf0db643fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:22.961 INFO 3009027 --- [ocp-schedules-10,368a8958d150df15,bee5780593f27a9b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:44:22.962 WARN 3009027 --- [ocp-schedules-10,368a8958d150df15,bee5780593f27a9b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:44:22.962 INFO 3009027 --- [ocp-schedules-10,368a8958d150df15,bee5780593f27a9b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:44:22.963 WARN 3009027 --- [ocp-schedules-10,368a8958d150df15,bee5780593f27a9b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:44:22.966 INFO 3009027 --- [ocp-schedules-10,368a8958d150df15,bee5780593f27a9b] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=2 2025-04-08 20:44:22.966 INFO 3009027 --- [ocp-schedules-20,e0044cff357cc273,091d65053c30d0c9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:44:22.966 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:44:22.966 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:44:22.968 ERROR 3009027 --- [ocp-schedules-26,4526a36a1affdf35,dc1f5acf0db643fa] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:44:22.968 INFO 3009027 --- [ocp-schedules-26,4526a36a1affdf35,dc1f5acf0db643fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 20:44:22.971 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:44:22.971 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:44:22.971 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:44:22.972 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:44:22.972 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:44:22.972 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:44:22.975 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:44:22.975 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:44:22.975 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:44:22.976 INFO 3009027 --- [ocp-schedules-27,8b756bbbcda19c53,f7f77584d877b49d] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:44:22.981 INFO 3009027 --- [ocp-schedules-14,629e3755e26ac0ee,7efb12c9b2262315] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:44:22.981+08:00, clusters=[2] 2025-04-08 20:44:22.981 INFO 3009027 --- [ocp-schedules-14,629e3755e26ac0ee,7efb12c9b2262315] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:44:22.985 INFO 3009027 --- [ocp-schedules-12,290b8481719db793,1ecfd3aac75815b0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:44:22.985 INFO 3009027 --- [ocp-schedules-12,290b8481719db793,1ecfd3aac75815b0] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:44:22.989 INFO 3009027 --- [pool-26-thread-1,290b8481719db793,719de949bc39d0ea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 20:44:22.993 INFO 3009027 --- [ocp-schedules-18,2012e4e22090be09,b8c3edb9634463e1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:44:22.995 WARN 3009027 --- [ocp-schedules-12,290b8481719db793,1ecfd3aac75815b0] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$ece00efc.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:44:22.995 INFO 3009027 --- [ocp-schedules-12,290b8481719db793,1ecfd3aac75815b0] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:44:23.000 INFO 3009027 --- [ocp-schedules-5,be6afbc45fa99fd0,2a8b66bcf0fb8877] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:44:23.000 INFO 3009027 --- [ocp-schedules-5,be6afbc45fa99fd0,2a8b66bcf0fb8877] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:44:23.004 WARN 3009027 --- [ocp-async-3,523dffa747118528,be4f5c7763ccf9f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:23.004 INFO 3009027 --- [ocp-async-7,3f476384b06b7b5e,8cf0beef33aa5d7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:23.006 INFO 3009027 --- [ocp-schedules-22,c5c3a648741b958f,5b1897d5452472d3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:44:23.006+08:00, clusters=[2] 2025-04-08 20:44:23.008 INFO 3009027 --- [ocp-schedules-22,c5c3a648741b958f,5b1897d5452472d3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:44:23.017 INFO 3009027 --- [ocp-schedules-28,19757bd3bc146092,0699f9725dab547e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:44:23.017 INFO 3009027 --- [ocp-schedules-28,19757bd3bc146092,0699f9725dab547e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:44:23.018 INFO 3009027 --- [ocp-schedules-8,aa0f3340586dfba8,744a3d9913e0c6a8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:44:23.020 INFO 3009027 --- [ocp-schedules-21,913d19f302e7f226,d2298a860df03047] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:44:23.021 INFO 3009027 --- [ocp-schedules-21,913d19f302e7f226,d2298a860df03047] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:23.022 INFO 3009027 --- [ocp-schedules-8,aa0f3340586dfba8,744a3d9913e0c6a8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:44:23.022 INFO 3009027 --- [ocp-schedules-8,aa0f3340586dfba8,744a3d9913e0c6a8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:44:23.022 INFO 3009027 --- [ocp-schedules-3,0d85d64cff9da306,d2d631038ceb585c] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 20:44:23.022 INFO 3009027 --- [ocp-schedules-3,0d85d64cff9da306,d2d631038ceb585c] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 20:44:23.022 INFO 3009027 --- [ocp-schedules-3,0d85d64cff9da306,d2d631038ceb585c] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=423 2025-04-08 20:44:23.024 INFO 3009027 --- [ocp-schedules-17,da5839441d6ce7b4,824cdb4b73fd92d2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:44:23.024 INFO 3009027 --- [ocp-schedules-17,da5839441d6ce7b4,824cdb4b73fd92d2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:44:23.029 INFO 3009027 --- [ocp-schedules-17,da5839441d6ce7b4,824cdb4b73fd92d2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:44:23.029 INFO 3009027 --- [ocp-schedules-17,da5839441d6ce7b4,824cdb4b73fd92d2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:44:23.032 INFO 3009027 --- [ocp-schedules-24,3c1206b65e32128a,2e605c87359d55aa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:44:23.032 ERROR 3009027 --- [ocp-schedules-21,913d19f302e7f226,d2298a860df03047] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:44:23.032 INFO 3009027 --- [ocp-schedules-21,913d19f302e7f226,d2298a860df03047] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:44:23.039 INFO 3009027 --- [ocp-schedules-3,0d85d64cff9da306,d2d631038ceb585c] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=423 2025-04-08 20:44:23.068 INFO 3009027 --- [ocp-schedules-29,a0a61eed302d6401,b4fa1e8241c1339a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:44:23.069 INFO 3009027 --- [ocp-schedules-29,a0a61eed302d6401,b4fa1e8241c1339a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:23.081 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:44:23.081 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:44:23.114 INFO 3009027 --- [ocp-schedules-15,e9710ad902522fa3,15c53ecbacea6682] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:44:23.128 INFO 3009027 --- [ocp-schedules-25,b494f3f6d1d18a13,8377958439b005ec] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:44:23.128 INFO 3009027 --- [ocp-schedules-25,b494f3f6d1d18a13,8377958439b005ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:44:23.128 INFO 3009027 --- [ocp-schedules-25,b494f3f6d1d18a13,8377958439b005ec] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:44:23.128 INFO 3009027 --- [ocp-schedules-25,b494f3f6d1d18a13,8377958439b005ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:44:23.128 INFO 3009027 --- [ocp-schedules-25,b494f3f6d1d18a13,8377958439b005ec] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:44:23.128 INFO 3009027 --- [ocp-schedules-25,b494f3f6d1d18a13,8377958439b005ec] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:44:23.161 INFO 3009027 --- [ocp-schedules-10,91a3c4c9929022e3,f431c2647d951c07] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:44:23.161 INFO 3009027 --- [ocp-schedules-10,91a3c4c9929022e3,f431c2647d951c07] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:44:23.175804+08:00, lastProcessTime=2025-04-08T20:44:13.176678+08:00 2025-04-08 20:44:23.177 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:44:23.178 ERROR 3009027 --- [ocp-schedules-10,91a3c4c9929022e3,f431c2647d951c07] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:44:23.178 INFO 3009027 --- [ocp-schedules-10,91a3c4c9929022e3,f431c2647d951c07] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:44:23.258 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:44:23.175804+08:00 2025-04-08 20:44:23.288 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:23.289 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:23.947 INFO 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:44:23.947 INFO 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:44:23.950 INFO 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:44:23.951 WARN 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:44:23.951 INFO 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:44:23.952 WARN 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:44:23.955 INFO 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:44:23.956 WARN 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:44:23.956 ERROR 3009027 --- [ocp-schedules-28,5fb34c54cea75dec,f6ae74569b351b76] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:44:24.176 INFO 3009027 --- [ocp-schedules-9,f6d10c3be5651cba,a7ee5771bb1abc30] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:44:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,8e48507909bfa0bf,c1cc915444096681] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:25.005 WARN 3009027 --- [ocp-async-1,6f9e7d61c5e558c1,492102b22786ec40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:25.005 INFO 3009027 --- [ocp-async-4,4581e30d8cbda6b0,9b637f23e58961e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:25.542 INFO 3009027 --- [ocp-schedules-21,039cc068e155b322,28f17c277287e5c8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:44:25.570 INFO 3009027 --- [ocp-schedules-21,039cc068e155b322,28f17c277287e5c8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:44:25.570 INFO 3009027 --- [ocp-schedules-21,039cc068e155b322,28f17c277287e5c8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:44:26.289 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:26.290 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:26.708 INFO 3009027 --- [ocp-schedules-7,2fa17e3d3c05f8fc,86fac92666497733] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:44:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:44:27.005 WARN 3009027 --- [ocp-async-4,f0cd30cdd9373e7a,a8e2bc6c967ab08a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:27.005 INFO 3009027 --- [ocp-async-10,39b4c505b3e9df85,8c855654978919c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,f21ab139f72b61b8,6f9380390101e6f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:28.726 INFO 3009027 --- [ocp-schedules-2,ef7aee6956a8bed8,483d5f2bb890a3b3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:44:28.727 INFO 3009027 --- [ocp-schedules-2,ef7aee6956a8bed8,483d5f2bb890a3b3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:44:29.005 WARN 3009027 --- [ocp-async-6,9ee27d745b9c2a94,be1a937f4b42eb59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:29.005 INFO 3009027 --- [ocp-async-3,e07e344338ff097e,3790b005bbaf1b9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:29.290 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:29.291 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,eac23aa59fe4b958,424a4d8abb3c2fa4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:31.005 WARN 3009027 --- [ocp-async-2,35052d6c8b0e5143,7d5b27d7b585d307] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:31.005 INFO 3009027 --- [ocp-async-1,100971149976839d,95a494693f255aa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:32.291 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:32.291 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:32.897 INFO 3009027 --- [ocp-schedules-19,187d96ff9881359f,503831f0ff6bc9c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:44:32.898 INFO 3009027 --- [ocp-schedules-1,ff44d96ca92ec8e2,312d2dc1cc9a83e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:44:32.928 INFO 3009027 --- [ocp-schedules-16,eecfe8330d9fb94b,545b7269c62311ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 20:44:32.948 INFO 3009027 --- [ocp-async-4,2e676948765c5c8f,19cae192d8d21408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:44:33.005 WARN 3009027 --- [ocp-async-10,a44b28ec58ba48a0,500360d1d1df7687] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:33.005 INFO 3009027 --- [ocp-async-1,aa9851ada7883e81,7e5b114ed264c941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:33.033 INFO 3009027 --- [ocp-schedules-21,cc2fd2c8ccfb3ee0,953d61633732584e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:44:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:44:33.176773+08:00, lastProcessTime=2025-04-08T20:44:23.175804+08:00 2025-04-08 20:44:33.262 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:44:33.176773+08:00 2025-04-08 20:44:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,6b4f8130fcb36d68,d1f9986a6d42180b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:35.005 INFO 3009027 --- [ocp-async-3,a7f4a85ddeee9ffe,5f5db92018442395] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:35.005 WARN 3009027 --- [ocp-async-2,0f46abbdaf720056,7e3b9cf086b4404a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:35.292 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:35.292 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,a7248957b7b28089,5aaa2989e9dd4de4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:36.737 INFO 3009027 --- [ocp-schedules-1,6000ce3e3ba91eaa,c0efa43798b03696] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:44:37.005 WARN 3009027 --- [ocp-async-8,2be4e888d6a296b2,b0a2f93b0b38a214] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:37.006 INFO 3009027 --- [ocp-async-4,c796557776a6cee3,37087d43b7e04abb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:38.292 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:38.293 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:38.375 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ec13127ad366f26,26ab81f1034a128f] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/8/retry, client 10.11.12.102, traceId 6ec13127ad366f26] 2025-04-08 20:44:38.379 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ec13127ad366f26,26ab81f1034a128f] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/8/retry?taskInstanceId=8, client=10.11.12.102, traceId=6ec13127ad366f26, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=8,] 2025-04-08 20:44:38.428 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ec13127ad366f26,26ab81f1034a128f] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 61, current state: FAILED, new state: READY 2025-04-08 20:44:38.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ec13127ad366f26,26ab81f1034a128f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/8/retry client=10.11.12.102, traceId=6ec13127ad366f26, duration=83 ms] 2025-04-08 20:44:38.455 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,6ec13127ad366f26,26ab81f1034a128f] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=8, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=null, startTime=2025-04-08T20:44:38.379+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 20:44:38.636 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,f344825fc3b3c4cf,8b8eb23ccbe61049] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=f344825fc3b3c4cf, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 20:44:38.636 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,e567826bf636daeb,5671800d08ecc9fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=e567826bf636daeb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:44:38.636 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-2,f344825fc3b3c4cf,8b8eb23ccbe61049] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=f344825fc3b3c4cf, duration=3 ms] 2025-04-08 20:44:38.645 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-4,e567826bf636daeb,5671800d08ecc9fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=e567826bf636daeb, duration=12 ms] 2025-04-08 20:44:38.732 INFO 3009027 --- [ocp-schedules-15,792164ae82a15310,85e0bdeca1f5eeb4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:44:38.733 INFO 3009027 --- [ocp-schedules-15,792164ae82a15310,85e0bdeca1f5eeb4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:44:39.006 WARN 3009027 --- [ocp-async-6,b31033d4d883c4c5,521fd46909937863] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:39.006 INFO 3009027 --- [ocp-async-10,e85c7b636ce84e66,bf02fb71d32654d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:39.012 INFO 3009027 --- [subtask-coordinator3,1308668168866431,6d5d90ef23038d50] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:RETRY, state: RUNNING 2025-04-08 20:44:39.036 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:44:39.053 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=61, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T20:44:39.008+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=61}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 20:44:39.060 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 20:44:39.066 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:44:39.070 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:44:39.139 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,0985fc8290972a9d,ac6c3089ff675355] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=0985fc8290972a9d, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 20:44:39.081 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:39.172 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,0985fc8290972a9d,ac6c3089ff675355] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=0985fc8290972a9d, duration=36 ms] 2025-04-08 20:44:39.187 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.277 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:39.297 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.337 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:44:39.364 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,2c4f857ec004b196,e26d86f6587c652f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:39.407 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.437 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:39.470 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.511 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:44:39.598 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.638 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.640 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:39.660 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.701 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 20:44:39.749 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.754 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.768 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 20:44:39.770 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:39.790 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:39.970 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,47e16fc01cebe357,8f8552b790d77f7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=47e16fc01cebe357, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:44:39.980 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-1,47e16fc01cebe357,8f8552b790d77f7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=47e16fc01cebe357, duration=13 ms] 2025-04-08 20:44:40.001 INFO 3009027 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:44:40.001 INFO 3009027 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:44:39.871 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 20:44:40.206 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,8503ca4c25ad7206,ec8e4b3b5cf37a73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=8503ca4c25ad7206, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 20:44:40.208 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,8503ca4c25ad7206,ec8e4b3b5cf37a73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=8503ca4c25ad7206, duration=5 ms] 2025-04-08 20:44:40.407 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:40.414 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:40.417 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 20:44:40.429 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 20:44:40.434 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 20:44:40.439 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:40.456 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:40.497 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 20:44:40.893 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:40.936 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:40.938 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 20:44:40.940 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 20:44:40.941 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 20:44:40.953 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:44:40.950 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 20:44:40.958 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(2), status=TAKINGOVER 2025-04-08 20:44:40.958 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 20:44:40.954 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:44:40.969 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:44:40.975 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:44:40.980 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:44:40.983 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 20:44:40.991 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 20:44:41.006 WARN 3009027 --- [ocp-async-2,529fe5e6a554adc1,ade049518ff1a884] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:41.006 INFO 3009027 --- [ocp-async-1,49a1d6b0f350073a,9a1325440ae08709] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:41.004 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:41.045 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.086 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:41.104 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.107 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:44:41.147 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.154 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.187 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:41.218 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,17e22b0d08650cd6,fafce5c3153c087a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=17e22b0d08650cd6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:44:41.226 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,17e22b0d08650cd6,fafce5c3153c087a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=17e22b0d08650cd6, duration=12 ms] 2025-04-08 20:44:41.206 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.274 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 20:44:41.293 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:41.294 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:41.305 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,f64f1cac6af61b99,37ea31e365fdfb58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=f64f1cac6af61b99, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 20:44:41.307 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,f64f1cac6af61b99,37ea31e365fdfb58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=f64f1cac6af61b99, duration=6 ms] 2025-04-08 20:44:41.327 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.329 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.338 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 20:44:41.385 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 20:44:41.440 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 20:44:41.445 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 20:44:41.477 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:44:41.490 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 20:44:41.535 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 20:44:41.539 ERROR 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 20:44:41.541 ERROR 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:44:41.570 WARN 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$9a1f85eb.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:44:41.583 INFO 3009027 --- [manual-subtask-executor16,24f1d56478c4d70e,d22165236fb9d394] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:RETRY, state: FAILED 2025-04-08 20:44:41.716 INFO 3009027 --- [task-coordinator1,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=8, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=2, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, createTime=2025-04-08T20:41:23+08:00, startTime=2025-04-08T20:41:23.075+08:00, endTime=2025-04-08T20:41:40.354+08:00, subtasks=[SubtaskInstanceEntity{id=76, name=Update OB cluster status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.obops.internal.cluster.task.UpdateObClusterStatusTask, seriesId=9, startTime=null, endTime=null}, SubtaskInstanceEntity{id=81, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=18, startTime=2025-04-08T20:41:25.317+08:00, endTime=2025-04-08T20:41:25.493+08:00}, SubtaskInstanceEntity{id=46, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=66, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=34, startTime=null, endTime=null}, SubtaskInstanceEntity{id=68, name=Active agent modules, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obagent.ActiveAgentModuleTask, seriesId=24, startTime=null, endTime=null}, SubtaskInstanceEntity{id=84, name=Wait node, state=PE 2025-04-08 20:44:41.731 WARN 3009027 --- [task-coordinator1,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:44:41.731 INFO 3009027 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8, status:FAILED 2025-04-08 20:44:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,e31489f5b3cae6a8,2fc3dc72e352c28c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:42.520 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,59d5c15a7dff12a3,ce268715aa78dac3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=59d5c15a7dff12a3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 20:44:42.528 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,59d5c15a7dff12a3,ce268715aa78dac3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=59d5c15a7dff12a3, duration=11 ms] 2025-04-08 20:44:42.613 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,2c15135ab80bea6f,25c3ffca204a1bd5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=2c15135ab80bea6f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 20:44:42.615 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,2c15135ab80bea6f,25c3ffca204a1bd5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=2c15135ab80bea6f, duration=5 ms] 2025-04-08 20:44:42.825 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,ae5ffbfb79155ad8,40fdd7c352252741] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=ae5ffbfb79155ad8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 20:44:42.842 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-3,ae5ffbfb79155ad8,40fdd7c352252741] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=ae5ffbfb79155ad8, duration=20 ms] 2025-04-08 20:44:42.948 INFO 3009027 --- [ocp-async-1,79472c7ba08e4d41,bf49b7b81582fb26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:44:43.006 WARN 3009027 --- [ocp-async-8,6e767445f667b514,aa88325e4b67d4db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:43.006 INFO 3009027 --- [ocp-async-5,62fc44706fa1b2d9,0bd711a765776265] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:43.035 INFO 3009027 --- [ocp-schedules-17,33fcf5973f3fe400,3562ac69b342c05d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:44:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:44:43.177135+08:00, lastProcessTime=2025-04-08T20:44:33.176773+08:00 2025-04-08 20:44:43.263 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:44:43.177135+08:00 2025-04-08 20:44:44.294 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:44.294 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:45.006 WARN 3009027 --- [ocp-async-6,edd563e481227919,436a148b5453c267] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:45.006 INFO 3009027 --- [ocp-async-10,68a275883a6f1d56,38d4229d0da60e26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,8f519d17b2fb161f,81e9963e05c4e00c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:45.572 INFO 3009027 --- [ocp-schedules-18,bd2ff47ce7ba8318,27c6770757b4dcd4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:44:45.577 INFO 3009027 --- [ocp-schedules-18,bd2ff47ce7ba8318,27c6770757b4dcd4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:44:45.577 INFO 3009027 --- [ocp-schedules-18,bd2ff47ce7ba8318,27c6770757b4dcd4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:44:46.758 INFO 3009027 --- [ocp-schedules-6,044b3f704d018481,e4e768e58b290c86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:44:47.006 WARN 3009027 --- [ocp-async-10,31c99d0b555459bf,0558710e2bb4485b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:47.006 INFO 3009027 --- [ocp-async-9,bf09510e79550f7a,44c086cd30902b11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:47.295 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:47.295 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,18cb24f397e50a5e,9173b43c1f03cce9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:48.736 INFO 3009027 --- [ocp-schedules-25,d37a44cce60b09be,fb9f34b18eb0aba9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:44:48.737 INFO 3009027 --- [ocp-schedules-25,d37a44cce60b09be,fb9f34b18eb0aba9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:44:49.006 INFO 3009027 --- [ocp-async-3,3a8b827032f66168,fe756d7cd2469d60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:49.006 WARN 3009027 --- [ocp-async-1,ba97ce876e976b9a,e7b3dcf64c863e9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:50.295 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:50.296 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:51.006 WARN 3009027 --- [ocp-async-3,5511eee3fff1baaa,1e42df3834583450] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:51.007 INFO 3009027 --- [ocp-async-6,2eaeb12fb7209e10,bf1b2443444e58a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,3a8a43315810b651,3fad9efca5c0f4cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:52.896 INFO 3009027 --- [ocp-schedules-23,b1a0688a18fea80b,b9ae44439be76f8c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:44:52.896 INFO 3009027 --- [ocp-schedules-6,8959022ca826bc41,e1a10298aeacff74] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:44:52.896 INFO 3009027 --- [ocp-schedules-24,ae1b9d5841d43196,e79a230ac76fa00b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:44:52.897 INFO 3009027 --- [ocp-schedules-6,8959022ca826bc41,e1a10298aeacff74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:52.897 INFO 3009027 --- [ocp-schedules-24,ae1b9d5841d43196,e79a230ac76fa00b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:52.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:44:52.899 INFO 3009027 --- [ocp-schedules-15,8ea1ae76981991dd,6677a4e0e59e16e6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:44:52.899 INFO 3009027 --- [ocp-schedules-15,8ea1ae76981991dd,6677a4e0e59e16e6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:44:52.899 INFO 3009027 --- [ocp-schedules-14,55cfa93eeebb5adb,105d32b1f70fa66f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:44:52.899 INFO 3009027 --- [ocp-schedules-14,55cfa93eeebb5adb,105d32b1f70fa66f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:44:52.899 INFO 3009027 --- [ocp-schedules-10,337c87b3a7346a98,d899d780506d452a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:44:52.899 INFO 3009027 --- [ocp-schedules-10,337c87b3a7346a98,d899d780506d452a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:44:52.911 INFO 3009027 --- [ocp-schedules-2,a82c3798412ed073,23c96541c94cd849] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:44:52.932 INFO 3009027 --- [ocp-schedules-4,e85f13f94b098d70,053045da5cfacfc3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:44:52.932 INFO 3009027 --- [ocp-schedules-4,d44ffb2a30208ae9,16dac49b437b20c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:44:52.933 INFO 3009027 --- [ocp-schedules-4,d44ffb2a30208ae9,16dac49b437b20c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:44:52.938 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:44:52.938 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:44:52.948 INFO 3009027 --- [ocp-async-2,74bb304580af92de,aa45e592be821c97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:44:52.995 INFO 3009027 --- [ocp-schedules-13,4d7e849c09b3116d,ab643273322e094c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:44:53.007 WARN 3009027 --- [ocp-async-7,7676d9c0c44213fb,2b1f31ffff0a0a08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:53.007 INFO 3009027 --- [ocp-async-6,6556dbf3c7771b34,15d29e762bed4525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:53.036 INFO 3009027 --- [ocp-schedules-7,6ab0cc4cdf75c7ec,fe6d3ecd265dd83d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:44:53.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:44:53.177594+08:00, lastProcessTime=2025-04-08T20:44:43.177135+08:00 2025-04-08 20:44:53.296 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:53.297 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:53.320 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:44:53.177594+08:00 2025-04-08 20:44:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,0bfbdffacb054bb0,d55a291fcca574aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:55.007 WARN 3009027 --- [ocp-async-1,35e9eeca32f92d19,04787690ad616c01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:55.007 INFO 3009027 --- [ocp-async-6,e65fc3bcad1f7e4a,fb001cd9c32d40a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:56.297 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:56.297 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:44:56.783 INFO 3009027 --- [ocp-schedules-18,fe011429f9857d90,c48f6f375aee5eae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:44:57.007 WARN 3009027 --- [ocp-async-6,3b6fef0d2ee7768d,5fef577f1ba79495] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:57.007 INFO 3009027 --- [ocp-async-3,77b105d76d1eb5c0,e2c4f0595de5e4ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,51ed6b5032d723ac,d5271452065bea6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116300, interval=60 2025-04-08 20:44:58.751 INFO 3009027 --- [ocp-schedules-10,8ba31df188ae049f,52317799de31df2f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:44:58.752 INFO 3009027 --- [ocp-schedules-10,8ba31df188ae049f,52317799de31df2f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:44:59.007 WARN 3009027 --- [ocp-async-7,c7ae5f388997ecc0,b346959a9a5a21c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:44:59.007 INFO 3009027 --- [ocp-async-3,60b01b7f47ecc501,82a176e0480a5f02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:44:59.297 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:44:59.298 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:00.054 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:40:00.0, nextRun=2025-04-08 20:45:00.0, currentDate=Tue Apr 08 20:45:00 CST 2025 2025-04-08 20:45:00.054 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:40:00.0, nextRunTime=2025-04-08 20:45:00.0} 2025-04-08 20:45:00.058 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 20:45:00.058 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T20:45:00.058+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='1308668168866431', operation=EXECUTE} 2025-04-08 20:45:00.063 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:40:00.0, nextRun=2025-04-08 20:45:00.0, currentDate=Tue Apr 08 20:45:00 CST 2025 2025-04-08 20:45:00.063 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:40:00.0, nextRunTime=2025-04-08 20:45:00.0} 2025-04-08 20:45:00.067 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 20:45:00.069 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 20:45:00.069 INFO 3009027 --- [task-coordinator2,1308668168866431,e7c23359eabb9bfe] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 20:45:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,15c538f5fa2e733d,b56ae1288b7a937d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:01.007 WARN 3009027 --- [ocp-async-1,fc8cdb3dd20a2f34,e0275c4d1dc588b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:01.007 INFO 3009027 --- [ocp-async-3,ecd30e1f9f2d2b89,0068fa57f609d574] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:01.059 INFO 3009027 --- [subtask-coordinator3,1308668168866431,c972206af7da8a72] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 85, operation:EXECUTE, state: RUNNING 2025-04-08 20:45:01.295 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 20:45:01.297 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=85, context=Context{parallelIdx=-1, stringMap={task_instance_id=9, task_operation=execute, latest_execution_start_time=2025-04-08T20:45:01.057+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=85}, listMap={}}, executor=192.168.1.61 2025-04-08 20:45:01.301 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.303 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 20:45:01.303 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.303 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.304 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.304 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.304 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.306 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.306 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 20:45:01.315 INFO 3009027 --- [subtask-executor7,1308668168866431,f66d91a90fbc0465] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 85, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 20:45:01.831 INFO 3009027 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9, status:SUCCESS 2025-04-08 20:45:01.831 INFO 3009027 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9, status:SUCCESS 2025-04-08 20:45:02.298 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:02.298 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:02.897 INFO 3009027 --- [ocp-schedules-6,8959022ca826bc41,e1a10298aeacff74] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:45:02.898 INFO 3009027 --- [ocp-schedules-24,ae1b9d5841d43196,e79a230ac76fa00b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:45:02.948 INFO 3009027 --- [ocp-async-3,64ff85a6717c91ae,846d2c9a5dc82e77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:45:03.007 WARN 3009027 --- [ocp-async-4,7195411423b64bc9,93c6b74d065ed233] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:03.007 INFO 3009027 --- [ocp-async-2,9d0a3c182d8c28d5,6a5d8c7758d7216b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:03.037 INFO 3009027 --- [ocp-schedules-30,1cc1b7f36dbbc782,65a770b1ffd70372] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:45:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:45:03.176076+08:00, lastProcessTime=2025-04-08T20:44:53.177594+08:00 2025-04-08 20:45:03.270 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:45:03.176076+08:00 2025-04-08 20:45:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,9b8944f3146b98f5,0d9dce6e0fa8ae5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:05.008 WARN 3009027 --- [ocp-async-10,05ced359bc35af2e,bd9de84ae7cac4e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:05.008 INFO 3009027 --- [ocp-async-9,20e55f84e598f3e8,8c69753e669273f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:05.299 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:05.299 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:05.578 INFO 3009027 --- [ocp-schedules-25,94b2f099d6ef2db4,41db9d3c4a2ea306] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:45:05.600 INFO 3009027 --- [ocp-schedules-25,94b2f099d6ef2db4,41db9d3c4a2ea306] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:45:05.600 INFO 3009027 --- [ocp-schedules-25,94b2f099d6ef2db4,41db9d3c4a2ea306] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:45:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,d3e4927cb76c1b18,7314d51964a86b49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:06.802 INFO 3009027 --- [ocp-schedules-17,ae8d7d0b9f1ff29f,46d2acbc8acddd9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:45:07.008 WARN 3009027 --- [ocp-async-9,9c147d6312d983ca,05da084a41fb967e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:07.008 INFO 3009027 --- [ocp-async-8,c72e363eb36b2ae4,748f604886b19b5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:08.299 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:08.300 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:08.830 INFO 3009027 --- [ocp-schedules-5,df72d13055e673b5,998a2a198268cdc3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:45:08.831 INFO 3009027 --- [ocp-schedules-5,df72d13055e673b5,998a2a198268cdc3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:45:09.008 WARN 3009027 --- [ocp-async-3,c6cb8abf32af7a40,447a366dbf89dff6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:09.008 INFO 3009027 --- [ocp-async-4,0639d50642f7ce97,22c9651d29924dcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,aeca84c706d1cccf,b73926ee39a59acf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:10.002 INFO 3009027 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:45:10.002 INFO 3009027 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:45:10.988 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:45:11.008 WARN 3009027 --- [ocp-async-4,caac503d6f407b80,ea3fa6e18ab63e36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:11.008 INFO 3009027 --- [ocp-async-2,06505500d544ea50,74b38f8284be606e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:11.300 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:11.301 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,4bfaa5e4fe7c3f29,b67688fbca30f489] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:12.408 INFO 3009027 --- [prometheus-cache-monitor1,2a15c2a015774792,5f2138d3ffef0a05] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":423,"secondWriteQueue":422,"minuteWriteCacheSize":420} 2025-04-08 20:45:12.408 INFO 3009027 --- [prometheus-cache-monitor1,2a15c2a015774792,5f2138d3ffef0a05] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:45:12.559 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T20:39:12.461, end=2025-04-08T20:44:12.515} 2025-04-08 20:45:12.559 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T20:39:12.46, end=2025-04-08T20:44:12.515} 2025-04-08 20:45:12.562 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 20:45:12.562 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 20:45:12.562 INFO 3009027 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:45:12.562 INFO 3009027 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 20:45:12.654 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116240, nodeCount=77, affectRows=77 2025-04-08 20:45:12.655 INFO 3009027 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116240, nodeCount=76, validCount=76 2025-04-08 20:45:12.655 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116240, nodeCount=84, affectRows=84 2025-04-08 20:45:12.655 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116240, nodeCount=89, affectRows=89 2025-04-08 20:45:12.656 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116240, nodeCount=78, affectRows=78 2025-04-08 20:45:12.656 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116240, nodeCount=94, affectRows=94 2025-04-08 20:45:12.656 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116240, nodeCount=84, validCount=84 2025-04-08 20:45:12.656 INFO 3009027 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116240, nodeCount=89, validCount=89 2025-04-08 20:45:12.656 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116240, nodeCount=77, validCount=77 2025-04-08 20:45:12.656 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116240, nodeCount=93, validCount=93 2025-04-08 20:45:12.948 INFO 3009027 --- [ocp-async-2,4c4a9960afc33e0d,cf860f3e62114ad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:45:13.008 WARN 3009027 --- [ocp-async-5,40a4119845341039,8598e3c5c0cf15db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:13.008 INFO 3009027 --- [ocp-async-8,f2d906619235f5c1,5b7fd0e6073d8061] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:13.039 INFO 3009027 --- [ocp-schedules-19,b3ae268b2c7442d8,83b9066b9f4f33f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:45:13.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:45:13.177222+08:00, lastProcessTime=2025-04-08T20:45:03.176076+08:00 2025-04-08 20:45:13.255 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:41:13.233, end=2025-04-08T20:43:13.253} 2025-04-08 20:45:13.257 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:45:13.257 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:45:13.258 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:45:13.177222+08:00 2025-04-08 20:45:14.301 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:14.301 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:15.008 WARN 3009027 --- [ocp-async-7,05cd3c8ee859b2e8,2bf166fa920c68f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:15.008 INFO 3009027 --- [ocp-async-8,2197393cded55c13,9e172702b3adf208] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,abf5401cd682297c,61c77967325bc5ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:16.826 INFO 3009027 --- [ocp-schedules-15,ff6e79de4c1c7c78,2ce9493301f39d74] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:45:17.009 WARN 3009027 --- [ocp-async-1,324de0757d1893a6,75e39bf0fcae5a45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:17.009 INFO 3009027 --- [ocp-async-8,f8dac7131fdf93e1,32b5407398015c75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:17.301 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:17.302 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,7a279542e969f876,aefb32c56050237f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:18.861 INFO 3009027 --- [ocp-schedules-23,2163ca656ec5a031,c364c93b5c84ab2e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:45:18.863 INFO 3009027 --- [ocp-schedules-23,2163ca656ec5a031,c364c93b5c84ab2e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:45:19.009 WARN 3009027 --- [ocp-async-5,61be6a3c4b4cd753,0988f697b1ff5098] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:19.009 INFO 3009027 --- [ocp-async-9,b14722654e007006,3953511a6d9dce78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:20.302 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:20.302 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:21.009 WARN 3009027 --- [ocp-async-6,7db8d5895c108a1f,1cbcc1a8171ac3f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:21.009 INFO 3009027 --- [ocp-async-7,4ceb03a773a5d17c,2cd942e8f8443801] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,13ea0686cb2f9196,e6aa250999ddbf48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:22.895 INFO 3009027 --- [ocp-schedules-11,e1eaf2a6855dda46,c82528a86ffb26a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:45:22.896 INFO 3009027 --- [ocp-schedules-20,7af8ea2e151d776f,e67c1b4bc8308212] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:45:22.896 INFO 3009027 --- [ocp-schedules-9,c97482907e299de6,7672ee0953860f8a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:45:22.896 INFO 3009027 --- [ocp-schedules-30,1b893dffac7780d8,05869768e0fd3856] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:45:22.896 INFO 3009027 --- [ocp-schedules-12,82b63b539bf3770a,e647a9b7fc1ea5e8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:45:22.896 INFO 3009027 --- [ocp-schedules-9,c97482907e299de6,7672ee0953860f8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:22.897 INFO 3009027 --- [ocp-schedules-12,82b63b539bf3770a,e647a9b7fc1ea5e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:45:22.898 INFO 3009027 --- [ocp-schedules-25,6323fb4385429ed0,fa4bcb212eeeb01f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:45:22.898 INFO 3009027 --- [ocp-schedules-24,8e76def8c32f4d5f,bf05ba101653cbd1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:45:22.898 INFO 3009027 --- [ocp-schedules-25,6323fb4385429ed0,fa4bcb212eeeb01f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:45:22.898 INFO 3009027 --- [ocp-schedules-24,8e76def8c32f4d5f,bf05ba101653cbd1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:45:22.898 INFO 3009027 --- [ocp-schedules-1,8995cb31cca18050,5e078540dd00db4b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:45:22.898 INFO 3009027 --- [ocp-schedules-1,8995cb31cca18050,5e078540dd00db4b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:45:22.902 INFO 3009027 --- [ocp-schedules-30,1b893dffac7780d8,05869768e0fd3856] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:45:22.921 INFO 3009027 --- [ocp-schedules-19,ac3d28cdd2344051,8d0af5d97bcb809b] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 20:45:22.921 INFO 3009027 --- [ocp-schedules-19,ac3d28cdd2344051,8d0af5d97bcb809b] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 20:45:22.923 INFO 3009027 --- [ocp-schedules-11,e1eaf2a6855dda46,c82528a86ffb26a6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:45:22.923 INFO 3009027 --- [ocp-schedules-2,973f85d63f9debb1,ac0b611fc095a888] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:45:22.938 INFO 3009027 --- [ocp-schedules-4,d44ffb2a30208ae9,16dac49b437b20c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:45:22.938 INFO 3009027 --- [ocp-schedules-4,a45551bce9f32a61,d5bde33168b5d817] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:45:22.939 INFO 3009027 --- [ocp-schedules-4,a45551bce9f32a61,d5bde33168b5d817] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:22.944 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:45:22.944 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:45:22.948 INFO 3009027 --- [ocp-async-1,a4ff256420acb6b2,c3f10f6f4694a90c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:45:22.949 INFO 3009027 --- [ocp-schedules-16,db3d56c0e90ff705,067a3d30601ede11] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:45:22.950 INFO 3009027 --- [ocp-schedules-16,db3d56c0e90ff705,067a3d30601ede11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:22.950 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:45:22.968 INFO 3009027 --- [ocp-schedules-3,c8f891a8399bea64,49b7d09066da5e12] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:45:22.986 INFO 3009027 --- [ocp-schedules-14,4191183286888303,b0e93f1da4991078] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:45:22.986+08:00, clusters=[2] 2025-04-08 20:45:22.987 INFO 3009027 --- [ocp-schedules-14,4191183286888303,b0e93f1da4991078] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:45:22.996 INFO 3009027 --- [ocp-schedules-20,e6303952ede3240d,682f4ffcfaa6fa99] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:45:23.003 INFO 3009027 --- [ocp-schedules-25,c8f8ffafdb773b52,c9dcda12bbf8a299] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:45:23.003 INFO 3009027 --- [ocp-schedules-25,c8f8ffafdb773b52,c9dcda12bbf8a299] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:45:23.009 WARN 3009027 --- [ocp-async-5,4797cda2002ae0b0,c7a9f815e3d20f24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:23.009 INFO 3009027 --- [ocp-async-10,d3c494363a0efce9,62758eea6e34d6d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:23.012 INFO 3009027 --- [ocp-schedules-24,12b0650ac00fed53,060c94c5e479cad7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:45:23.012+08:00, clusters=[2] 2025-04-08 20:45:23.014 INFO 3009027 --- [ocp-schedules-24,12b0650ac00fed53,060c94c5e479cad7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:45:23.019 INFO 3009027 --- [ocp-schedules-21,ea1faa8018dd2d3c,15de1b68d6bb0a69] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:45:23.019 INFO 3009027 --- [ocp-schedules-21,ea1faa8018dd2d3c,15de1b68d6bb0a69] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:45:23.022 INFO 3009027 --- [ocp-schedules-19,ca8398d81b60c9c8,0e1e2dc899c0c897] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:45:23.024 INFO 3009027 --- [ocp-schedules-19,ca8398d81b60c9c8,0e1e2dc899c0c897] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:45:23.024 INFO 3009027 --- [ocp-schedules-19,ca8398d81b60c9c8,0e1e2dc899c0c897] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:45:23.029 INFO 3009027 --- [ocp-schedules-18,165d0214a34d0c88,6ddc1f95c11d3bea] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:45:23.029 INFO 3009027 --- [ocp-schedules-18,165d0214a34d0c88,6ddc1f95c11d3bea] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:45:23.033 INFO 3009027 --- [ocp-schedules-18,165d0214a34d0c88,6ddc1f95c11d3bea] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:45:23.033 INFO 3009027 --- [ocp-schedules-18,165d0214a34d0c88,6ddc1f95c11d3bea] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:45:23.040 INFO 3009027 --- [ocp-schedules-6,d6b12e281e3deaf8,96a4b198e02731d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:45:23.087 INFO 3009027 --- [ocp-schedules-29,a0a61eed302d6401,b4fa1e8241c1339a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:45:23.120 INFO 3009027 --- [ocp-schedules-23,1edfc625873e3af5,7583e0ed329eba50] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:45:23.129 INFO 3009027 --- [ocp-schedules-7,4421118762f6ca6a,6a7dffc94dc5d126] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:45:23.129 INFO 3009027 --- [ocp-schedules-7,4421118762f6ca6a,6a7dffc94dc5d126] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:45:23.129 INFO 3009027 --- [ocp-schedules-7,4421118762f6ca6a,6a7dffc94dc5d126] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:45:23.129 INFO 3009027 --- [ocp-schedules-7,4421118762f6ca6a,6a7dffc94dc5d126] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:45:23.129 INFO 3009027 --- [ocp-schedules-7,4421118762f6ca6a,6a7dffc94dc5d126] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:45:23.129 INFO 3009027 --- [ocp-schedules-7,4421118762f6ca6a,6a7dffc94dc5d126] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:45:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:45:23.175633+08:00, lastProcessTime=2025-04-08T20:45:13.177222+08:00 2025-04-08 20:45:23.178 INFO 3009027 --- [ocp-schedules-15,b3435455a4230c21,ff1cfa0e2a25d45b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:45:23.179 INFO 3009027 --- [ocp-schedules-15,b3435455a4230c21,ff1cfa0e2a25d45b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:23.194 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:45:23.195 ERROR 3009027 --- [ocp-schedules-15,b3435455a4230c21,ff1cfa0e2a25d45b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:45:23.195 INFO 3009027 --- [ocp-schedules-15,b3435455a4230c21,ff1cfa0e2a25d45b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:45:23.257 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:45:23.175633+08:00 2025-04-08 20:45:23.303 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:23.303 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:23.960 INFO 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:45:23.960 INFO 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:45:23.963 INFO 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:45:23.964 WARN 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:45:23.964 INFO 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:45:23.965 WARN 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:45:23.968 INFO 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:45:23.969 WARN 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:45:23.970 ERROR 3009027 --- [ocp-schedules-1,b8aab3fac4208bb1,5b07a4fef2eb060f] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:45:24.277 INFO 3009027 --- [ocp-schedules-18,852cd3d044af18a9,37940d630a33d4b3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:45:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,0f22ce9cf7d08c8c,eefe9c32e1289c03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:25.009 INFO 3009027 --- [ocp-async-10,90e40c75ac632718,77bfde32e9cecfb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:25.009 WARN 3009027 --- [ocp-async-6,2ac24234dcc80ed0,13a45a89126e9140] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:25.601 INFO 3009027 --- [ocp-schedules-21,cae4cfc7174dbe52,cdf18baa070b8f24] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:45:25.604 INFO 3009027 --- [ocp-schedules-21,cae4cfc7174dbe52,cdf18baa070b8f24] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:45:25.604 INFO 3009027 --- [ocp-schedules-21,cae4cfc7174dbe52,cdf18baa070b8f24] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:45:26.303 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:26.304 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:26.846 INFO 3009027 --- [ocp-schedules-15,894bb26bfb0d9a50,c62b9dc054aab4f3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:45:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:45:27.009 WARN 3009027 --- [ocp-async-10,4db8e6db1ce52380,667b4791592a90a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:27.009 INFO 3009027 --- [ocp-async-7,8e980470b0add355,f850c68c86fa220f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,b918052b406e825f,19ff7f6ddfd89586] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:28.919 INFO 3009027 --- [ocp-schedules-30,c55533f9af525fcc,e0c7aa7a312fb560] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:45:28.920 INFO 3009027 --- [ocp-schedules-30,c55533f9af525fcc,e0c7aa7a312fb560] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:45:29.010 WARN 3009027 --- [ocp-async-1,6eb605a12b85bc39,25d6639e81daa10b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:29.010 INFO 3009027 --- [ocp-async-8,08b0c4ddc0b64eb6,18b196c5087ee68d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:29.304 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:29.304 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,beeece7ac41292e1,43825683f63a0252] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:31.010 INFO 3009027 --- [ocp-async-8,aebf67694508cdae,f05217214816d5f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:31.010 WARN 3009027 --- [ocp-async-3,cacb313398411603,de07ab8833cabe9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:31.692 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:45:31.692 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:45:31.695 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:45:31.695 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:45:31.695 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:45:31.695 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:45:31.699 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:45:31.699 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:45:31.699 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:45:31.699 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:45:31.720 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:45:31.726 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:45:31.726 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:45:31.726 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:45:31.726 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:45:31.726 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:45:31.731 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:45:32.050 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:45:32.304 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:32.305 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:32.365 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.666202694S 2025-04-08 20:45:32.365 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:45:32.422 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:45:32.422 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:45:32.447 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:45:32.486 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:45:32.486 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:45:32.486 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:45:32.486 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:45:32.486 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:45:32.486 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:45:32.493 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:45:32.593 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] o.s.boot.SpringApplication : Started application in 0.897 seconds (JVM running for 1374.21) 2025-04-08 20:45:32.593 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.897642725S 2025-04-08 20:45:32.856 INFO 3009027 --- [ocp-schedules-25,c91cf5ef18bb3b9b,0856a4210d4b8c96] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:45:32.897 INFO 3009027 --- [ocp-schedules-9,c97482907e299de6,7672ee0953860f8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:45:32.897 INFO 3009027 --- [ocp-schedules-12,82b63b539bf3770a,e647a9b7fc1ea5e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:45:32.949 INFO 3009027 --- [ocp-async-4,668cfa560dca86cb,ba37ea71968ef709] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:45:32.951 INFO 3009027 --- [ocp-schedules-16,db3d56c0e90ff705,067a3d30601ede11] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:45:33.010 WARN 3009027 --- [ocp-async-2,7122def3dc41946c,4632d3ba249c99b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:33.010 INFO 3009027 --- [ocp-async-7,5faca292043311dc,5e93278a1ac81204] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:33.041 INFO 3009027 --- [ocp-schedules-3,3da39e3629b3cf7f,66f33602be555366] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:45:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:45:33.176814+08:00, lastProcessTime=2025-04-08T20:45:23.175633+08:00 2025-04-08 20:45:33.319 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:45:33.176814+08:00 2025-04-08 20:45:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,8b5a611cf077acc0,9b51333f4cb1e5db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:35.010 WARN 3009027 --- [ocp-async-7,1f90600c6c84a02e,216a49b5c37154ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:35.010 INFO 3009027 --- [ocp-async-3,8ba6a8a55435e9c4,7fc65c32a27db7e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:35.305 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:35.306 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,e26260c25f60363b,dee42a367a51aa93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:36.879 INFO 3009027 --- [ocp-schedules-29,de36c463fb7d8d3b,63ffc70aad8c9477] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:45:37.010 WARN 3009027 --- [ocp-async-8,cde54a5e7d5cf648,faeb3842a700a5ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:37.010 INFO 3009027 --- [ocp-async-10,ada13ecd12ff4e92,2e23be48fe641ae2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:38.306 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:38.306 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:38.934 INFO 3009027 --- [ocp-schedules-13,941b8448bcd704c0,a38160c67997d784] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:45:38.935 INFO 3009027 --- [ocp-schedules-13,941b8448bcd704c0,a38160c67997d784] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:45:39.010 WARN 3009027 --- [ocp-async-2,6a84eb3fc833afce,dacb4aa42cdde5d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:39.010 INFO 3009027 --- [ocp-async-5,57d154e0bfce3069,f920b73130cf62eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,f7c6dfc156b1147c,6880596984a34843] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:40.001 INFO 3009027 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:45:40.001 INFO 3009027 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:45:41.010 WARN 3009027 --- [ocp-async-4,6edbad8e9b87d073,3d311f778c483d9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:41.011 INFO 3009027 --- [ocp-async-6,a40727f3daee5c56,6f2a76be25963ad5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:41.306 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:41.307 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,e7a1442f0c764a57,603f24e7188a1ccf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:42.949 INFO 3009027 --- [ocp-async-6,db1b92baf76fdd27,23aaaddff98b1fb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:45:43.011 WARN 3009027 --- [ocp-async-3,3420bba7130e8aa6,53b02e613ab40167] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:43.011 INFO 3009027 --- [ocp-async-10,cd6332a121a73e8b,a4133fa35af6c2b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:43.042 INFO 3009027 --- [ocp-schedules-17,d0edfcf4139e19b3,18f22694b88e9cfe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:45:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:45:43.176690+08:00, lastProcessTime=2025-04-08T20:45:33.176814+08:00 2025-04-08 20:45:43.268 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:45:43.176690+08:00 2025-04-08 20:45:44.307 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:44.308 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:45.011 WARN 3009027 --- [ocp-async-2,05a6ae4f62dfb12e,be6a3a49724b821a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:45.011 INFO 3009027 --- [ocp-async-5,139c42b8449f747f,1ae03bd85cfd8775] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,f51c24b56aea66e2,633eb088299e4645] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:45.606 INFO 3009027 --- [ocp-schedules-15,14b8c05943738fb7,6ca501bb7aa72324] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:45:45.703 INFO 3009027 --- [ocp-schedules-15,14b8c05943738fb7,6ca501bb7aa72324] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:45:45.703 INFO 3009027 --- [ocp-schedules-15,14b8c05943738fb7,6ca501bb7aa72324] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:45:46.900 INFO 3009027 --- [ocp-schedules-26,a93a73d096123f4d,70e2d34ddc7f1427] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:45:47.011 WARN 3009027 --- [ocp-async-9,b85d75bb0d1ed4b2,c21c36fabe4e6917] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:47.011 INFO 3009027 --- [ocp-async-7,4c821f18c4887935,f6f0d7c9844f86f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:47.308 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:47.308 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,6266dd319af76930,7cdb90f981ce8167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:48.938 INFO 3009027 --- [ocp-schedules-3,73265cba1efe7cf2,79846089ed9edef3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:45:48.939 INFO 3009027 --- [ocp-schedules-3,73265cba1efe7cf2,79846089ed9edef3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:45:49.011 INFO 3009027 --- [ocp-async-8,1ea84b9a23bb06a4,ce8c9e2821268e37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:49.011 WARN 3009027 --- [ocp-async-6,a4da177547383654,dbe91435a8a4569d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:50.308 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:50.309 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:51.011 WARN 3009027 --- [ocp-async-2,092e03949a580060,72243f2d726878ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:51.011 INFO 3009027 --- [ocp-async-1,b567c74c78d63514,352614404d1d2bec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,93a4ca6ba60df0bb,7087937c48048107] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:52.896 INFO 3009027 --- [ocp-schedules-25,65428a419f432050,5d932d4e780112f2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:45:52.896 INFO 3009027 --- [ocp-schedules-19,0fbab114898853b2,e9f97e344d15da05] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:45:52.896 INFO 3009027 --- [ocp-schedules-7,efe153db10d387ae,3010d6f93974a528] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:45:52.896 INFO 3009027 --- [ocp-schedules-19,0fbab114898853b2,e9f97e344d15da05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:52.897 INFO 3009027 --- [ocp-schedules-7,efe153db10d387ae,3010d6f93974a528] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:45:52.898 INFO 3009027 --- [ocp-schedules-23,191b0172b61f1ab5,d844e09442e44dc0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:45:52.898 INFO 3009027 --- [ocp-schedules-23,191b0172b61f1ab5,d844e09442e44dc0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:45:52.898 INFO 3009027 --- [ocp-schedules-15,49509fec6de1f0cd,39bd8a4613d269c2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:45:52.898 INFO 3009027 --- [ocp-schedules-15,49509fec6de1f0cd,39bd8a4613d269c2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:45:52.899 INFO 3009027 --- [ocp-schedules-16,f343a84daecdba46,57aad7339ad1f750] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:45:52.899 INFO 3009027 --- [ocp-schedules-16,f343a84daecdba46,57aad7339ad1f750] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:45:52.900 INFO 3009027 --- [ocp-schedules-9,10a1a23d674edcbc,df924abf85ec8b56] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:45:52.944 INFO 3009027 --- [ocp-schedules-4,a45551bce9f32a61,d5bde33168b5d817] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:45:52.945 INFO 3009027 --- [ocp-schedules-4,687178db6728e062,75e674c4cd1c92f3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:45:52.945 INFO 3009027 --- [ocp-schedules-4,687178db6728e062,75e674c4cd1c92f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:45:52.949 INFO 3009027 --- [ocp-async-1,fc8b1f714774002f,5823bb98be473a8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:45:52.950 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:45:52.950 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:45:52.997 INFO 3009027 --- [ocp-schedules-6,3bb8af7dc6941b5a,e27d2d31756f68ee] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:45:53.011 WARN 3009027 --- [ocp-async-9,6bb5b52f95582615,414cbeb0fede148d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:53.011 INFO 3009027 --- [ocp-async-4,4ac5ee0b31c60ab3,671d80e0cbd1b9f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:53.044 INFO 3009027 --- [ocp-schedules-29,bd4b4954f33e8c88,ad4c80d7276a0ba4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:45:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:45:53.176555+08:00, lastProcessTime=2025-04-08T20:45:43.176690+08:00 2025-04-08 20:45:53.275 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:45:53.176555+08:00 2025-04-08 20:45:53.309 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:53.310 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:53.624 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-7,f38591e252de11bd,39b3df572a73c3bb] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=f38591e252de11bd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744116353621014,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 20:45:53.626 WARN 3009027 --- [http-nio-0.0.0.0-8080-exec-7,f38591e252de11bd,39b3df572a73c3bb] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=f38591e252de11bd, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744116353621014,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=4 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 20:45:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,fc8844081d35a986,4de7c4e2b72b4eaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:55.012 WARN 3009027 --- [ocp-async-4,360c7dec8cde82fe,be3c4e60bedd378c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:55.012 INFO 3009027 --- [ocp-async-3,a4fdb1c5d4500cee,b8002f7793a216cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:56.310 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:56.311 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:45:56.928 INFO 3009027 --- [ocp-schedules-15,8f3a59e55bdae8bf,05614713694bb7a4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:45:57.012 WARN 3009027 --- [ocp-async-2,1bd3c94207dc2707,451f78e716d14868] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:57.012 INFO 3009027 --- [ocp-async-10,ea6fb460ad500dcd,3b2d1e6b7ca9fca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,c444320f9f3512c1,a89249c890affd46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116360, interval=60 2025-04-08 20:45:58.946 INFO 3009027 --- [ocp-schedules-11,30adec61e1414471,10b759978819a117] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:45:58.948 INFO 3009027 --- [ocp-schedules-11,30adec61e1414471,10b759978819a117] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:45:59.012 WARN 3009027 --- [ocp-async-9,e93f27569f631cb4,bb6d1433416f5943] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:45:59.012 INFO 3009027 --- [ocp-async-7,02ed613254b00fcf,a082867ca62949c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:45:59.311 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:45:59.311 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,5137418e0fdfa9cd,3057ee528d8046c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:01.012 WARN 3009027 --- [ocp-async-5,2f921af27054d8dd,b33e2236f8fd2753] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:01.012 INFO 3009027 --- [ocp-async-4,16776861a468b953,ad62c15e45911481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:02.312 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:02.312 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:02.897 INFO 3009027 --- [ocp-schedules-19,0fbab114898853b2,e9f97e344d15da05] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:46:02.898 INFO 3009027 --- [ocp-schedules-7,efe153db10d387ae,3010d6f93974a528] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:46:02.949 INFO 3009027 --- [ocp-async-8,65c7c90ba6abcaab,af3042aff20aca59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:46:03.012 WARN 3009027 --- [ocp-async-1,32403eade1f54ca9,ce7e2d2ff0711f69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:03.012 INFO 3009027 --- [ocp-async-2,58d0365d69560034,4bb2d67318139b00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:03.045 INFO 3009027 --- [ocp-schedules-10,f3a80d77072d067b,b725b941d42d1388] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:46:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:46:03.176766+08:00, lastProcessTime=2025-04-08T20:45:53.176555+08:00 2025-04-08 20:46:03.297 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:46:03.176766+08:00 2025-04-08 20:46:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,74bd48e9cafa7ebf,f8594871dbf4b74c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:05.012 WARN 3009027 --- [ocp-async-10,4e4c374c7c7dc9d3,259152cf1d2fe780] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:05.012 INFO 3009027 --- [ocp-async-2,35399aff33ad9df2,9f61a4fc9c21c980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:05.312 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:05.313 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:05.705 INFO 3009027 --- [ocp-schedules-14,1afdd94f72711680,0076b624d9f005ad] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:46:05.741 INFO 3009027 --- [ocp-schedules-14,1afdd94f72711680,0076b624d9f005ad] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:46:05.741 INFO 3009027 --- [ocp-schedules-14,1afdd94f72711680,0076b624d9f005ad] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:46:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,ab50972354dea0af,e78db5667bfde9dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:06.948 INFO 3009027 --- [ocp-schedules-25,24b000d0fbc7d5c1,e5a2874c1ca82061] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:46:07.013 WARN 3009027 --- [ocp-async-2,1c06118dd7aa98d2,986d36b360df5e62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:07.013 INFO 3009027 --- [ocp-async-6,cf2610bab582b839,6f200018eebb0c1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:08.313 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:08.313 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:08.997 INFO 3009027 --- [ocp-schedules-7,995ab29029b23110,575bbe34247adafe] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:46:08.998 INFO 3009027 --- [ocp-schedules-7,995ab29029b23110,575bbe34247adafe] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:46:09.013 WARN 3009027 --- [ocp-async-1,b4388701187a4d23,24bde8f65e10cbdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:09.013 INFO 3009027 --- [ocp-async-3,a7011412390c59e5,6e96236b275a6bf2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,16988a0ccd5b4bde,48a51c64c21ff445] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:10.002 INFO 3009027 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:46:10.002 INFO 3009027 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:46:10.991 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:46:11.013 WARN 3009027 --- [ocp-async-10,56a3eed1b19dd56f,4b7e892c9d3d7c67] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:11.013 INFO 3009027 --- [ocp-async-7,bf4b0a0b9d867553,252b930078aca946] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:11.313 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:11.314 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,57dac1b7bae20fd0,5877848b5fb65872] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:12.408 INFO 3009027 --- [prometheus-cache-monitor1,8c02a6e749380279,5b0e931115d523d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":423,"secondWriteQueue":422,"minuteWriteCacheSize":420} 2025-04-08 20:46:12.408 INFO 3009027 --- [prometheus-cache-monitor1,8c02a6e749380279,5b0e931115d523d1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:46:12.666 INFO 3009027 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116300, nodeCount=70, affectRows=70 2025-04-08 20:46:12.666 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116300, nodeCount=153, affectRows=153 2025-04-08 20:46:12.666 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116300, nodeCount=80, affectRows=80 2025-04-08 20:46:12.666 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116300, nodeCount=56, affectRows=56 2025-04-08 20:46:12.667 INFO 3009027 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116300, nodeCount=63, affectRows=63 2025-04-08 20:46:12.667 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116300, nodeCount=56, validCount=56 2025-04-08 20:46:12.667 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116300, nodeCount=79, validCount=79 2025-04-08 20:46:12.667 INFO 3009027 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116300, nodeCount=62, validCount=62 2025-04-08 20:46:12.667 INFO 3009027 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116300, nodeCount=70, validCount=70 2025-04-08 20:46:12.667 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116300, nodeCount=152, validCount=152 2025-04-08 20:46:12.949 INFO 3009027 --- [ocp-async-9,e17f7a1b92b340cb,2d5ac2125ebbad9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:46:13.013 WARN 3009027 --- [ocp-async-2,756d7e458bcbf0d7,75254bab77bdbc56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:13.013 INFO 3009027 --- [ocp-async-4,edf6e42206842f73,d43214c0c8da29d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:13.047 INFO 3009027 --- [ocp-schedules-24,fa2f5ce0ec4d03fa,da5674a79e373ac6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:46:13.179 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:46:13.178252+08:00, lastProcessTime=2025-04-08T20:46:03.176766+08:00 2025-04-08 20:46:13.285 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:46:13.178252+08:00 2025-04-08 20:46:14.314 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:14.315 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:15.013 WARN 3009027 --- [ocp-async-1,197349455fdf2fbe,90a48ced6a2fc5cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:15.013 INFO 3009027 --- [ocp-async-8,9137c5f049801243,28abc3edb51785dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,d5aa5ef4b3c9a280,b37212f1c9046144] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:16.977 INFO 3009027 --- [ocp-schedules-14,3441be0a5f1e5ba1,38c0ca0c95137d45] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:46:17.013 WARN 3009027 --- [ocp-async-10,206dac8d6bc859a6,0de28b943ddef324] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:17.013 INFO 3009027 --- [ocp-async-7,bd03e89281443cd6,1da9207db9412867] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:17.315 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:17.316 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,0c7deeb54b6f5b13,667eb171646ee22b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:19.004 INFO 3009027 --- [ocp-schedules-8,7e929c26a441ec6a,f4542581456cc62c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:46:19.005 INFO 3009027 --- [ocp-schedules-8,7e929c26a441ec6a,f4542581456cc62c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:46:19.013 WARN 3009027 --- [ocp-async-6,46f020a252f7929b,a60a6b14ac8d963e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:19.014 INFO 3009027 --- [ocp-async-9,11e6bfbfbab9a082,3633129c8884e518] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:20.316 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:20.316 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:21.014 WARN 3009027 --- [ocp-async-1,e54e425048fe3a5a,ee03997dd4162f9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:21.014 INFO 3009027 --- [ocp-async-4,0c03194b7b4dd89f,aa342d7eac21a155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,9df7529d7899b365,1c279d504d591644] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:22.891 INFO 3009027 --- [ocp-schedules-11,38ef37785b3fd5b1,4c0e44b887755c0e] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T20:46:22.891+08:00, tenantSize=0 2025-04-08 20:46:22.895 INFO 3009027 --- [ocp-schedules-22,409387f2a45e482e,0e12c499849c5a82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:46:22.896 INFO 3009027 --- [ocp-schedules-28,9becffd62e03a244,b9f27dc4abf9b0b3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:46:22.896 INFO 3009027 --- [ocp-schedules-14,65cb4e034fbc49e8,d5b76e99a83793bb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:46:22.896 INFO 3009027 --- [ocp-schedules-5,7d58646257710b76,4e63cbe686331301] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:46:22.896 INFO 3009027 --- [ocp-schedules-14,65cb4e034fbc49e8,d5b76e99a83793bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:22.896 INFO 3009027 --- [ocp-schedules-29,ed3474004517ffdb,4fb1969c5c626f7f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:46:22.897 INFO 3009027 --- [ocp-schedules-29,ed3474004517ffdb,4fb1969c5c626f7f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:22.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:46:22.898 INFO 3009027 --- [ocp-schedules-6,d3932b72e6366060,aaf7e850b7f354ef] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:46:22.898 INFO 3009027 --- [ocp-schedules-6,d3932b72e6366060,aaf7e850b7f354ef] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:46:22.898 INFO 3009027 --- [ocp-schedules-21,6822bb6f19d634ba,4c5ad6ae9bd29d9f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:46:22.898 INFO 3009027 --- [ocp-schedules-21,6822bb6f19d634ba,4c5ad6ae9bd29d9f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:46:22.898 INFO 3009027 --- [ocp-schedules-9,6294b94a5552d40f,c357cda353a2425d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:46:22.898 INFO 3009027 --- [ocp-schedules-9,6294b94a5552d40f,c357cda353a2425d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:46:22.902 INFO 3009027 --- [ocp-schedules-7,1e45c5aa573b87f5,cc794f59cf08dca6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:46:22.902 INFO 3009027 --- [ocp-schedules-22,409387f2a45e482e,0e12c499849c5a82] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:46:22.902 INFO 3009027 --- [ocp-schedules-5,7d58646257710b76,4e63cbe686331301] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:46:22.915 INFO 3009027 --- [ocp-schedules-30,20f6b7386824888f,27588b521f9861ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":420,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:46:22.915 INFO 3009027 --- [ocp-schedules-30,20f6b7386824888f,27588b521f9861ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:46:22.920 INFO 3009027 --- [ocp-schedules-15,b33a600149040547,d74ef740d9ee4c09] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:46:22.921 INFO 3009027 --- [ocp-schedules-15,b33a600149040547,d74ef740d9ee4c09] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 20:46:22.936 INFO 3009027 --- [ocp-schedules-2,4a3ce3fcd9957fa1,fcef63be9fce6d64] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 20:46:22.936 INFO 3009027 --- [ocp-schedules-2,4a3ce3fcd9957fa1,fcef63be9fce6d64] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 20:46:22.950 INFO 3009027 --- [ocp-async-10,266a0943686904e3,62426261c04adb15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:46:22.951 INFO 3009027 --- [ocp-schedules-4,687178db6728e062,75e674c4cd1c92f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:46:22.951 INFO 3009027 --- [ocp-schedules-4,be675ed488c74725,061e34a7a776cbe6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:46:22.951 INFO 3009027 --- [ocp-schedules-4,be675ed488c74725,061e34a7a776cbe6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:22.956 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:46:22.956 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:46:22.969 INFO 3009027 --- [ocp-schedules-10,24909f86aff588d0,4be4bec9eded913c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:46:22.992 INFO 3009027 --- [ocp-schedules-13,c3c93031a1936cd1,fd049f69ba8b5b39] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:46:22.992+08:00, clusters=[2] 2025-04-08 20:46:22.993 INFO 3009027 --- [ocp-schedules-13,c3c93031a1936cd1,fd049f69ba8b5b39] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:46:22.998 INFO 3009027 --- [ocp-schedules-20,381e81150cf1fcc2,47382a6c4bcb4736] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:46:23.005 INFO 3009027 --- [ocp-schedules-28,6ea7f2359a2c02ff,335490cad8969555] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:46:23.005 INFO 3009027 --- [ocp-schedules-28,6ea7f2359a2c02ff,335490cad8969555] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:46:23.014 WARN 3009027 --- [ocp-async-7,cf794b6a961e0ba0,8b74b571da2b18f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:23.014 INFO 3009027 --- [ocp-async-6,73690254ee5d53dd,3917327f1875a87d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:23.019 INFO 3009027 --- [ocp-schedules-5,2c7ac93ba546b107,bf80f75f63e1f533] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:46:23.019+08:00, clusters=[2] 2025-04-08 20:46:23.021 INFO 3009027 --- [ocp-schedules-7,6cb98c1addc6eee2,b760c25ed15da402] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:46:23.021 INFO 3009027 --- [ocp-schedules-7,6cb98c1addc6eee2,b760c25ed15da402] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:46:23.022 INFO 3009027 --- [ocp-schedules-5,2c7ac93ba546b107,bf80f75f63e1f533] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:46:23.024 INFO 3009027 --- [ocp-schedules-22,b72d38051354234b,7ee1c8869fbaf0fe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:46:23.025 INFO 3009027 --- [ocp-schedules-22,b72d38051354234b,7ee1c8869fbaf0fe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:46:23.025 INFO 3009027 --- [ocp-schedules-22,b72d38051354234b,7ee1c8869fbaf0fe] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:46:23.033 INFO 3009027 --- [ocp-schedules-30,19b4a83d34067160,3e7f0e33716bdf78] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:46:23.033 INFO 3009027 --- [ocp-schedules-30,19b4a83d34067160,3e7f0e33716bdf78] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:23.033 INFO 3009027 --- [ocp-schedules-17,dd96c9f6a1017c09,f62fb09d11c2b42e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:46:23.034 INFO 3009027 --- [ocp-schedules-17,dd96c9f6a1017c09,f62fb09d11c2b42e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:46:23.038 INFO 3009027 --- [ocp-schedules-17,dd96c9f6a1017c09,f62fb09d11c2b42e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:46:23.038 INFO 3009027 --- [ocp-schedules-17,dd96c9f6a1017c09,f62fb09d11c2b42e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:46:23.044 ERROR 3009027 --- [ocp-schedules-30,19b4a83d34067160,3e7f0e33716bdf78] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:46:23.044 INFO 3009027 --- [ocp-schedules-30,19b4a83d34067160,3e7f0e33716bdf78] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:46:23.048 INFO 3009027 --- [ocp-schedules-15,f737f94588218ab0,60f5da883f068d19] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:46:23.087 INFO 3009027 --- [ocp-schedules-27,dfd29da0511f046a,a00b980cf34a9be6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:46:23.088 INFO 3009027 --- [ocp-schedules-27,dfd29da0511f046a,a00b980cf34a9be6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:23.099 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:23.099 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:46:23.126 INFO 3009027 --- [ocp-schedules-19,384c48981b8d14e4,a3c0e4d1ada4b387] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:46:23.130 INFO 3009027 --- [ocp-schedules-8,4b39648519692259,722ab48011d4771e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:46:23.130 INFO 3009027 --- [ocp-schedules-8,4b39648519692259,722ab48011d4771e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:46:23.132 INFO 3009027 --- [ocp-schedules-8,4b39648519692259,722ab48011d4771e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:46:23.132 INFO 3009027 --- [ocp-schedules-8,4b39648519692259,722ab48011d4771e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:46:23.135 INFO 3009027 --- [ocp-schedules-8,4b39648519692259,722ab48011d4771e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:46:23.135 INFO 3009027 --- [ocp-schedules-8,4b39648519692259,722ab48011d4771e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:46:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:46:23.176111+08:00, lastProcessTime=2025-04-08T20:46:13.178252+08:00 2025-04-08 20:46:23.195 INFO 3009027 --- [ocp-schedules-24,46dfd941c66aa6b4,1b73fe7ac6417bdf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:46:23.196 INFO 3009027 --- [ocp-schedules-24,46dfd941c66aa6b4,1b73fe7ac6417bdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:23.212 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:46:23.213 ERROR 3009027 --- [ocp-schedules-24,46dfd941c66aa6b4,1b73fe7ac6417bdf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:46:23.213 INFO 3009027 --- [ocp-schedules-24,46dfd941c66aa6b4,1b73fe7ac6417bdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:46:23.301 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:46:23.176111+08:00 2025-04-08 20:46:23.316 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:23.317 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:23.973 INFO 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:46:23.973 INFO 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:46:23.976 INFO 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:46:23.977 WARN 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:46:23.977 INFO 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:46:23.978 WARN 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:46:23.981 INFO 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:46:23.983 WARN 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:46:23.987 ERROR 3009027 --- [ocp-schedules-22,9014cbe931a1bcb4,e12920cee8f9811a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:46:24.302 INFO 3009027 --- [ocp-schedules-2,aa1c0782e258e2d2,ca6394fa8f7c5a00] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:46:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,2791d039fc53b7aa,b02065a3b369b904] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:25.014 WARN 3009027 --- [ocp-async-9,96e568c4d339faf9,d492eac8605ed917] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:25.014 INFO 3009027 --- [ocp-async-1,234761c886de4fc0,5a5802b8fbd4abbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:25.743 INFO 3009027 --- [ocp-schedules-3,42a94706318aa69e,f30b76ad07c58b81] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:46:25.748 INFO 3009027 --- [ocp-schedules-3,42a94706318aa69e,f30b76ad07c58b81] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:46:25.748 INFO 3009027 --- [ocp-schedules-3,42a94706318aa69e,f30b76ad07c58b81] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:46:26.317 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:26.318 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:46:27.000 INFO 3009027 --- [ocp-schedules-11,71ae7964e6e54749,db23bb29199074e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:46:27.014 INFO 3009027 --- [ocp-async-5,32a4fc8b018373ad,0915932c66491305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:27.014 WARN 3009027 --- [ocp-async-1,b11b259c8eb6263e,511018a30517df79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,bfde2e08828401bb,9b4dcf588ae0ca91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:29.014 WARN 3009027 --- [ocp-async-2,ae0fe1bcb6f2a9b5,9d8c8fbc94e2b1d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:29.014 INFO 3009027 --- [ocp-async-7,89d53d5f575e3152,72e59ade1340ae0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:29.078 INFO 3009027 --- [ocp-schedules-1,c613ebf64c830b54,688b8de7e9f20a2c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:46:29.079 INFO 3009027 --- [ocp-schedules-1,c613ebf64c830b54,688b8de7e9f20a2c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:46:29.318 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:29.319 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,87b1f6690ea4d35d,b8558c3937b12543] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:31.014 WARN 3009027 --- [ocp-async-7,0ed16953ac955330,62967b2a41fd033e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:31.014 INFO 3009027 --- [ocp-async-4,fe15c647e67acde1,817600403a1271a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:32.319 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:32.319 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:32.897 INFO 3009027 --- [ocp-schedules-14,65cb4e034fbc49e8,d5b76e99a83793bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:46:32.898 INFO 3009027 --- [ocp-schedules-29,ed3474004517ffdb,4fb1969c5c626f7f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:46:32.950 INFO 3009027 --- [ocp-async-8,51afe6535542d5be,25914b2b93037c18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:46:32.951 INFO 3009027 --- [ocp-schedules-1,cd8822ca6da856c5,5269a23838b8cce8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:46:32.951 INFO 3009027 --- [ocp-schedules-1,cd8822ca6da856c5,5269a23838b8cce8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:32.952 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:46:33.015 WARN 3009027 --- [ocp-async-5,1c12034b5f871edd,418d3e3043853c85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:33.015 INFO 3009027 --- [ocp-async-3,497c757ac2904853,3ad158ff9dd1b70e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:33.049 INFO 3009027 --- [ocp-schedules-25,e15f63a6cf3707b8,b40d41c25118b217] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:46:33.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:46:33.175939+08:00, lastProcessTime=2025-04-08T20:46:23.176111+08:00 2025-04-08 20:46:33.275 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:46:33.175939+08:00 2025-04-08 20:46:33.347 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 20:46:33.347 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 20:46:33.348 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.348 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:46:33.349 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.349 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.349 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.349 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 20:46:33.359 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.359 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.359 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.360 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 20:46:33.401 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 20:46:33.401 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 20:46:33.402 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.402 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:46:33.403 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.403 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.403 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.403 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 20:46:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,0e0fe8dd95288935,ce35c4840a9e654c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:33.415 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.415 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.415 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.415 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 20:46:33.426 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.426 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.427 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.427 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 20:46:33.439 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.440 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.440 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.440 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 20:46:33.451 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.451 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.452 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.452 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 20:46:33.457 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.457 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.457 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.457 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 20:46:33.461 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.461 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.461 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.461 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 20:46:33.464 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.465 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.465 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.465 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 20:46:33.469 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.469 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.470 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.470 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 20:46:33.473 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.474 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.474 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.474 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 20:46:33.478 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.478 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.478 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.479 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 20:46:33.485 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.485 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.486 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.486 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 20:46:33.496 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.496 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.497 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.497 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 20:46:33.501 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.501 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.502 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.502 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 20:46:33.507 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.507 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.507 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.508 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 20:46:33.512 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.512 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.512 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.513 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 20:46:33.516 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.516 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.516 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.516 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 20:46:33.520 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.521 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.521 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.521 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 20:46:33.526 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.526 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.526 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.526 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 20:46:33.530 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.531 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.531 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.531 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 20:46:33.534 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.535 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.535 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.535 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 20:46:33.538 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.538 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.539 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.539 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 20:46:33.542 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.542 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.542 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.543 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 20:46:33.546 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.546 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.546 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.547 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 20:46:33.549 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.550 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.550 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.550 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 20:46:33.553 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.554 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.554 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.554 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 20:46:33.557 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.557 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.558 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.558 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 20:46:33.561 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.562 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.562 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.562 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 20:46:33.566 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.566 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.566 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.566 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 20:46:33.570 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.570 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.570 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.570 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 20:46:33.573 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.574 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.574 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.574 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 20:46:33.578 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.578 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.578 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.579 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 20:46:33.582 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.582 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.582 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.582 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 20:46:33.585 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.585 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.585 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.586 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 20:46:33.591 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.591 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.591 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.591 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 20:46:33.596 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.596 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.596 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.596 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 20:46:33.601 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.601 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.601 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.601 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 20:46:33.606 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.606 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.606 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.607 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 20:46:33.609 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.610 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.610 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.610 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 20:46:33.613 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.613 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.613 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.613 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 20:46:33.616 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.616 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.616 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.616 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 20:46:33.619 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.619 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.620 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.620 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 20:46:33.628 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.628 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.628 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.628 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 20:46:33.647 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.648 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.648 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.648 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 20:46:33.653 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.653 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.654 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.654 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 20:46:33.659 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.659 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.659 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.660 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 20:46:33.665 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.666 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.667 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.667 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 20:46:33.670 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.670 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.671 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.671 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 20:46:33.674 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.674 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 20:46:33.674 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:33.674 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 20:46:33.677 INFO 3009027 --- [ocp-schedules-30,e593521d46b4a251,12640fd24b4ad8ee] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 20:46:35.015 WARN 3009027 --- [ocp-async-7,ed9cfc2dc239d2e7,a6d3004f71192675] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:35.015 INFO 3009027 --- [ocp-async-10,a84e6c6ebb6ab403,b0dbf1ade5a04407] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:35.319 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:35.320 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,d42387603a85da1a,dd9dab10093a4e9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:37.015 WARN 3009027 --- [ocp-async-10,fe169ec0b1fbd1bd,b84aaf83c3d7628a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:37.015 INFO 3009027 --- [ocp-async-4,fc04f2e532d11858,d0751a4d3121dbb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:37.024 INFO 3009027 --- [ocp-schedules-3,7b68bd029bdd0322,bd4f0e6211faf838] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:46:38.320 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:38.321 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:39.015 WARN 3009027 --- [ocp-async-2,8ceea9c9eb1ed719,e77fc9f62101d361] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:39.015 INFO 3009027 --- [ocp-async-4,b9ab4ee99ba55969,accc358152156096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:39.197 INFO 3009027 --- [ocp-schedules-29,54d912b08e6443cc,5e797a036c258c60] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:46:39.198 INFO 3009027 --- [ocp-schedules-29,54d912b08e6443cc,5e797a036c258c60] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:46:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,b685e6d50b1f0d70,d5195184f8f22126] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:40.001 INFO 3009027 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:46:40.001 INFO 3009027 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:46:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:40.971 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:46:40.972 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:46:40.972 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:46:40.972 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:46:40.979 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(2), status=TAKINGOVER 2025-04-08 20:46:40.979 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=7ms, period=120000ms 2025-04-08 20:46:41.015 WARN 3009027 --- [ocp-async-4,b3b7d27f42b6fbb1,398d5477549eca03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:41.015 INFO 3009027 --- [ocp-async-6,8514bcbc5b5045f9,483f5430b6f26b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:41.321 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:41.322 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,56d672baea766bb9,98175fdc377e439e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:42.950 INFO 3009027 --- [ocp-async-6,931d2c0dedb3d5c4,de2a6b3516f9608e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:46:42.952 INFO 3009027 --- [ocp-schedules-1,cd8822ca6da856c5,5269a23838b8cce8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:46:43.015 WARN 3009027 --- [ocp-async-9,63ca0a6bc2c852f0,b98330b76d3213f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:43.015 INFO 3009027 --- [ocp-async-10,fbf213e9b19dd854,ed5b60b9a5e107f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:43.051 INFO 3009027 --- [ocp-schedules-16,e93be1ae6402ea0a,9489ee4ef33aa443] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:46:43.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:46:43.177300+08:00, lastProcessTime=2025-04-08T20:46:33.175939+08:00 2025-04-08 20:46:43.266 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:46:43.177300+08:00 2025-04-08 20:46:44.322 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:44.322 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:45.015 WARN 3009027 --- [ocp-async-2,4a8dbbb952eb52de,74f3c6554ab020bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:45.016 INFO 3009027 --- [ocp-async-1,6573536fb3e8e64c,642e29fbcabc78ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,de7e72f1248cd583,e18062ddd9fde1e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:45.750 INFO 3009027 --- [ocp-schedules-17,e4f1bebc65be90a6,8ed9c0575e220246] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:46:45.762 INFO 3009027 --- [ocp-schedules-17,e4f1bebc65be90a6,8ed9c0575e220246] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:46:45.762 INFO 3009027 --- [ocp-schedules-17,e4f1bebc65be90a6,8ed9c0575e220246] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:46:47.016 INFO 3009027 --- [ocp-async-1,250423df49d356c9,2338ef6ea948fa80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:47.016 WARN 3009027 --- [ocp-async-7,3d5c5a5c8497d26a,33b312d1359b4184] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:47.323 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:47.323 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:47.357 INFO 3009027 --- [ocp-schedules-2,ad59c21532dd83bb,85da8d41084ee35d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:46:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,cf53496e35b0f64b,b2d2c5349e869237] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:49.016 WARN 3009027 --- [ocp-async-5,9e8ce35511e10d8b,65e6acb844183532] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:49.016 INFO 3009027 --- [ocp-async-1,9d97096145635c5b,77c050fac49bbcb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:49.219 INFO 3009027 --- [ocp-schedules-24,38786c6921ebad00,9c8ed2f6f27bf3bf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:46:49.220 INFO 3009027 --- [ocp-schedules-24,38786c6921ebad00,9c8ed2f6f27bf3bf] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:46:50.323 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:50.324 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:51.016 WARN 3009027 --- [ocp-async-3,6a1679d0cc3e86c7,5d36ba3e8ccd268f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:51.016 INFO 3009027 --- [ocp-async-1,0022fee5855cff70,0703b94de405e299] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,48847216c04bb4f4,37d78d7321ca3a12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:52.896 INFO 3009027 --- [ocp-schedules-24,28eda0c23257befd,acd48ced910d87c1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:46:52.896 INFO 3009027 --- [ocp-schedules-15,22850535d827b400,26607befe343f540] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:46:52.896 INFO 3009027 --- [ocp-schedules-15,22850535d827b400,26607befe343f540] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:52.896 INFO 3009027 --- [ocp-schedules-3,afd013bc312e9835,9418819b8ed6d224] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:46:52.897 INFO 3009027 --- [ocp-schedules-3,afd013bc312e9835,9418819b8ed6d224] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:52.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:46:52.899 INFO 3009027 --- [ocp-schedules-5,ed2fe357ef22d674,228e3d1e0af53781] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:46:52.899 INFO 3009027 --- [ocp-schedules-5,ed2fe357ef22d674,228e3d1e0af53781] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:46:52.899 INFO 3009027 --- [ocp-schedules-28,2bc24166291b10b6,524a496fc14313d4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:46:52.899 INFO 3009027 --- [ocp-schedules-28,2bc24166291b10b6,524a496fc14313d4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:46:52.899 INFO 3009027 --- [ocp-schedules-14,bc7874cbc85d8fd8,cfd93faf92b5150f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:46:52.899 INFO 3009027 --- [ocp-schedules-14,bc7874cbc85d8fd8,cfd93faf92b5150f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:46:52.909 INFO 3009027 --- [ocp-schedules-22,28963dd340b3194c,b501890a24866640] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:46:52.950 INFO 3009027 --- [ocp-async-10,f09297a7615aee0b,54162195cdfa5b95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:46:52.956 INFO 3009027 --- [ocp-schedules-4,be675ed488c74725,061e34a7a776cbe6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:46:52.956 INFO 3009027 --- [ocp-schedules-4,2e8172c5fc03e475,8f905f595b15f3dc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:46:52.957 INFO 3009027 --- [ocp-schedules-4,2e8172c5fc03e475,8f905f595b15f3dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:46:52.962 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:46:52.962 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:46:53.000 INFO 3009027 --- [ocp-schedules-19,93222e47790f5ba5,49c42d199e01ed51] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:46:53.016 WARN 3009027 --- [ocp-async-7,b2f663bd8805fdc2,6036a740b6b381f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:53.016 INFO 3009027 --- [ocp-async-6,32d47c1bac2a563d,b2f9ea369b11986f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:53.053 INFO 3009027 --- [ocp-schedules-25,833508e95f2b30b7,a1cdf88e972799ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:46:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:46:53.177438+08:00, lastProcessTime=2025-04-08T20:46:43.177300+08:00 2025-04-08 20:46:53.263 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:46:53.177438+08:00 2025-04-08 20:46:53.324 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:53.324 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,6e04348ef0f4ff5a,ad1f00307353b6ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:55.016 WARN 3009027 --- [ocp-async-9,2b72757eca2fd1f1,a05dc40c2119c15f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:55.016 INFO 3009027 --- [ocp-async-6,2a6a468362334df8,720d33377d44eb93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:56.325 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:56.325 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:46:57.016 WARN 3009027 --- [ocp-async-2,74c8c8a31c1ccbda,e6d5aca0c042d02e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:57.016 INFO 3009027 --- [ocp-async-1,5d730540a6f037a0,79f35cc477f084b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:57.381 INFO 3009027 --- [ocp-schedules-24,4653370f3e326e21,39182228468c6942] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:46:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,595cbf0840b36b83,b631b65acec84379] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116420, interval=60 2025-04-08 20:46:59.017 WARN 3009027 --- [ocp-async-4,e91eb6080f2d9f8c,09e04f04936b8c15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:46:59.017 INFO 3009027 --- [ocp-async-7,5d2922d8a1a441e5,0ece43101e9bcba2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:46:59.299 INFO 3009027 --- [ocp-schedules-17,ad0fd2f7e5e9d210,a50c9f6ff6e314f8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:46:59.300 INFO 3009027 --- [ocp-schedules-17,ad0fd2f7e5e9d210,a50c9f6ff6e314f8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:46:59.325 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:46:59.326 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,b56fba6b88f6833f,239e5e3f71ea6f34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:01.017 WARN 3009027 --- [ocp-async-9,85f8a198aed36618,e100a051d4752cae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:01.017 INFO 3009027 --- [ocp-async-8,1ce0a4901051a516,e5761eb0916d8231] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:02.326 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:02.326 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:02.897 INFO 3009027 --- [ocp-schedules-15,22850535d827b400,26607befe343f540] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:47:02.898 INFO 3009027 --- [ocp-schedules-3,afd013bc312e9835,9418819b8ed6d224] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:47:02.950 INFO 3009027 --- [ocp-async-8,b8e9b580ba238701,c434f58f0ae14a77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:47:03.017 WARN 3009027 --- [ocp-async-2,9eb0b77139a9dcfb,ab45ecfa0de99e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:03.017 INFO 3009027 --- [ocp-async-3,912f7f1af6071178,e7f82bb1ac81c452] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:03.054 INFO 3009027 --- [ocp-schedules-2,5320095081b74712,713bf82da1b65c42] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:47:03.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:47:03.176242+08:00, lastProcessTime=2025-04-08T20:46:53.177438+08:00 2025-04-08 20:47:03.264 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:47:03.176242+08:00 2025-04-08 20:47:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,1abbe279fdd531bb,f32637785b34ad60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:05.017 WARN 3009027 --- [ocp-async-5,fc1a54d0db6c8b7a,c90483842d6094fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:05.017 INFO 3009027 --- [ocp-async-1,742f54454ae8384a,be6780cac0dcef93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:05.327 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:05.327 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:05.763 INFO 3009027 --- [ocp-schedules-20,8c21cf6d92cdd7ea,d5e1657520b3b2ac] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:47:05.775 INFO 3009027 --- [ocp-schedules-20,8c21cf6d92cdd7ea,d5e1657520b3b2ac] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:47:05.775 INFO 3009027 --- [ocp-schedules-20,8c21cf6d92cdd7ea,d5e1657520b3b2ac] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:47:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,6bb146b8babb18c6,886e5c8d669968a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:07.017 WARN 3009027 --- [ocp-async-6,0c9b9fe22a979ebd,240dd84238cb8f0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:07.017 INFO 3009027 --- [ocp-async-1,c63eaa89a37c77a9,6bd4a0bf247eeb25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:07.402 INFO 3009027 --- [ocp-schedules-2,36f1575f601b2651,541958e106f246c7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:47:08.327 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:08.328 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:09.017 WARN 3009027 --- [ocp-async-2,df5818e335624f79,81b0a8e3436fb7dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:09.017 INFO 3009027 --- [ocp-async-1,95ad716a42335063,d0757b3f443d38b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:09.309 INFO 3009027 --- [ocp-schedules-8,d312d43124d21a79,6899de96c2770271] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:47:09.310 INFO 3009027 --- [ocp-schedules-8,d312d43124d21a79,6899de96c2770271] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:47:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,b1fab644d7a97454,a884cfcf3f17bc05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:10.002 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:47:10.002 INFO 3009027 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:47:10.988 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:47:11.017 WARN 3009027 --- [ocp-async-5,d547e0ca0dbc355c,8943be421f2664d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:11.018 INFO 3009027 --- [ocp-async-3,4b5af58f88046fde,0eaa20cd68a2a644] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:11.328 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:11.328 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,29fa134e9f20d00c,e0968ceacac9d786] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:12.408 INFO 3009027 --- [prometheus-cache-monitor1,65b6801adc7620d9,17b46a4419e3010f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":423,"secondWriteQueue":422,"minuteWriteCacheSize":420} 2025-04-08 20:47:12.408 INFO 3009027 --- [prometheus-cache-monitor1,65b6801adc7620d9,17b46a4419e3010f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:47:12.681 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116360, nodeCount=38, affectRows=38 2025-04-08 20:47:12.681 INFO 3009027 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116360, nodeCount=37, validCount=37 2025-04-08 20:47:12.683 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116360, nodeCount=93, affectRows=93 2025-04-08 20:47:12.683 INFO 3009027 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116360, nodeCount=95, affectRows=95 2025-04-08 20:47:12.683 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116360, nodeCount=91, affectRows=91 2025-04-08 20:47:12.683 INFO 3009027 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116360, nodeCount=92, validCount=92 2025-04-08 20:47:12.683 INFO 3009027 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116360, nodeCount=95, validCount=95 2025-04-08 20:47:12.683 INFO 3009027 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116360, nodeCount=91, validCount=91 2025-04-08 20:47:12.684 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116360, nodeCount=105, affectRows=105 2025-04-08 20:47:12.685 INFO 3009027 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116360, nodeCount=104, validCount=104 2025-04-08 20:47:12.950 INFO 3009027 --- [ocp-async-9,bbaf0e65ad98abae,5a5ead0df70cf56c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:47:13.018 WARN 3009027 --- [ocp-async-10,a25ed0a83e2e663b,059e98ff088a8940] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:13.018 INFO 3009027 --- [ocp-async-7,6ebc5d6c220ebcc8,e7db2f2bca46fc49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:13.055 INFO 3009027 --- [ocp-schedules-22,778168ba385cb577,ef96a2cf899d4b18] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:47:13.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:47:13.175616+08:00, lastProcessTime=2025-04-08T20:47:03.176242+08:00 2025-04-08 20:47:13.261 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:47:13.175616+08:00 2025-04-08 20:47:13.268 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:43:13.253, end=2025-04-08T20:45:13.266} 2025-04-08 20:47:13.272 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 20:47:13.272 INFO 3009027 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 20:47:14.329 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:14.329 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:15.018 WARN 3009027 --- [ocp-async-8,950d67b992a0f9c2,b0d67d1250811b9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:15.018 INFO 3009027 --- [ocp-async-2,35ab01f01374e2aa,c988fb6ca74c45b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,7bbbafd4d6b1a892,3169736f7c79d73b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:16.225 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5da5649018f426e8,8058dc2106166364] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/50/logSegments?taskInstanceId=8&subtaskInstanceId=50, client=10.11.12.102, traceId=5da5649018f426e8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,50,] 2025-04-08 20:47:16.314 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-10,5da5649018f426e8,8058dc2106166364] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/50/logSegments client=10.11.12.102, traceId=5da5649018f426e8, duration=173 ms] 2025-04-08 20:47:17.018 WARN 3009027 --- [ocp-async-5,0701ab2741bfb790,438cc5a3c798a4f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:17.018 INFO 3009027 --- [ocp-async-9,e8350b8ffaedf4f7,7008829e9494d822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:17.027 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,e9cc610e0f1a1969,46f443e079ca9a2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/81/logSegments?taskInstanceId=8&subtaskInstanceId=81, client=10.11.12.102, traceId=e9cc610e0f1a1969, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,81,] 2025-04-08 20:47:17.029 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-8,e9cc610e0f1a1969,46f443e079ca9a2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/81/logSegments client=10.11.12.102, traceId=e9cc610e0f1a1969, duration=6 ms] 2025-04-08 20:47:17.329 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:17.330 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:17.430 INFO 3009027 --- [ocp-schedules-12,1a726f8aaf255460,5963229ab55460a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:47:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,8d43b8680610e2d4,1ee5c5798a19cbdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:19.018 WARN 3009027 --- [ocp-async-3,ef89b8679133078c,b8939202170f6eb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:19.018 INFO 3009027 --- [ocp-async-6,7fcf8311a9ca18e1,2d4af24a758a81a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:19.316 INFO 3009027 --- [ocp-schedules-16,947e741c98f7a4e5,cea97d140673cb45] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:47:19.317 INFO 3009027 --- [ocp-schedules-16,947e741c98f7a4e5,cea97d140673cb45] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:47:20.330 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:20.331 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:21.018 WARN 3009027 --- [ocp-async-7,b1955e4b5427c840,888d4423b20e5cac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:21.018 INFO 3009027 --- [ocp-async-4,dc9e73a439720c5b,567923bcca1c724e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,8b40069687b0d357,d8b575beedcd281d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:22.895 INFO 3009027 --- [ocp-schedules-11,fe752b838430e668,5d835dc0daa88c21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:47:22.896 INFO 3009027 --- [ocp-schedules-6,a00d22e8598e6ba4,f89f72330f6bac6c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:47:22.896 INFO 3009027 --- [ocp-schedules-12,2c038747bbb2087a,0605c9ea08142c40] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:47:22.896 INFO 3009027 --- [ocp-schedules-12,2c038747bbb2087a,0605c9ea08142c40] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:22.897 INFO 3009027 --- [ocp-schedules-15,98c1193226fdebec,ddca928be5ef38b8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:47:22.897 INFO 3009027 --- [ocp-schedules-15,98c1193226fdebec,ddca928be5ef38b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:22.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-28,0dc900cfcddddeae,c828327218d571fe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-22,1271eb392e1c88ab,90ccb67e9ac1d93f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-22,1271eb392e1c88ab,90ccb67e9ac1d93f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-18,12c6d4b672c91352,ded77f44af28da38] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-24,c706fbc51d1b84bc,fe797fb9eda9c84b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-24,c706fbc51d1b84bc,fe797fb9eda9c84b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:47:22.898 INFO 3009027 --- [ocp-schedules-18,12c6d4b672c91352,ded77f44af28da38] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:47:22.904 INFO 3009027 --- [ocp-schedules-28,0dc900cfcddddeae,c828327218d571fe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:47:22.950 INFO 3009027 --- [ocp-async-5,51630a94146b1e56,ff993c083378aadd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:47:22.962 INFO 3009027 --- [ocp-schedules-4,2e8172c5fc03e475,8f905f595b15f3dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:47:22.963 INFO 3009027 --- [ocp-schedules-4,bdcbf535eec50fd0,fd0d230054ad6cc5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:47:22.963 INFO 3009027 --- [ocp-schedules-4,bdcbf535eec50fd0,fd0d230054ad6cc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:22.968 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:47:22.968 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:47:22.970 INFO 3009027 --- [ocp-schedules-21,419d5a96d195fe69,2ee17a35fbcc484e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:47:22.971 INFO 3009027 --- [ocp-schedules-9,3877d13025ad682f,6354d0f107e26094] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:47:22.971 INFO 3009027 --- [ocp-schedules-11,fe752b838430e668,5d835dc0daa88c21] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:47:22.977 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 20:47:22.977 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:47:22.982 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:47:22.982 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 20:47:22.982 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:47:22.984 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:47:22.984 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 20:47:22.984 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:47:22.986 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:47:22.986 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 20:47:22.986 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:47:22.988 INFO 3009027 --- [ocp-schedules-1,373f30faa29643e8,400b9a255adae41f] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:47:22.998 INFO 3009027 --- [ocp-schedules-5,432101ad540922c6,1f10a2bbb2617399] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:47:22.998+08:00, clusters=[2] 2025-04-08 20:47:22.998 INFO 3009027 --- [ocp-schedules-5,432101ad540922c6,1f10a2bbb2617399] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:47:23.001 INFO 3009027 --- [ocp-schedules-10,593462985576b29a,af496911edb10535] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:47:23.007 INFO 3009027 --- [ocp-schedules-22,1e09b66fe5b2d276,55deefd2ce341e9f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:47:23.007 INFO 3009027 --- [ocp-schedules-22,1e09b66fe5b2d276,55deefd2ce341e9f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:47:23.018 WARN 3009027 --- [ocp-async-10,b09265f989e0107d,924097618efe4326] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:23.018 INFO 3009027 --- [ocp-async-3,989477fd82e0aa92,3108174a2f5ae7a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:23.023 INFO 3009027 --- [ocp-schedules-8,a02fbb8b7cf9b6fb,5b679df6b2ddf99c] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:47:23.023 INFO 3009027 --- [ocp-schedules-8,a02fbb8b7cf9b6fb,5b679df6b2ddf99c] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:47:23.025 INFO 3009027 --- [ocp-schedules-30,9924c010ba68e0e8,59e05fc415c13c80] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:47:23.026 INFO 3009027 --- [ocp-schedules-28,d98b8758ca52a475,98f99ea06315582b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:47:23.026+08:00, clusters=[2] 2025-04-08 20:47:23.027 INFO 3009027 --- [ocp-schedules-28,d98b8758ca52a475,98f99ea06315582b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:47:23.028 INFO 3009027 --- [ocp-schedules-30,9924c010ba68e0e8,59e05fc415c13c80] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:47:23.028 INFO 3009027 --- [ocp-schedules-30,9924c010ba68e0e8,59e05fc415c13c80] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:47:23.039 INFO 3009027 --- [ocp-schedules-14,2eea3680ba895d77,d8293a74c1537fc1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:47:23.039 INFO 3009027 --- [ocp-schedules-14,2eea3680ba895d77,d8293a74c1537fc1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:47:23.043 INFO 3009027 --- [ocp-schedules-14,2eea3680ba895d77,d8293a74c1537fc1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:47:23.043 INFO 3009027 --- [ocp-schedules-14,2eea3680ba895d77,d8293a74c1537fc1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:47:23.057 INFO 3009027 --- [ocp-schedules-3,8308447a10c8b4c3,b0153334264259ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:47:23.058 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e00faddbeeae65bd,110fcb6b6a726619] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/47/logSegments?taskInstanceId=8&subtaskInstanceId=47, client=10.11.12.102, traceId=e00faddbeeae65bd, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,47,] 2025-04-08 20:47:23.060 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-5,e00faddbeeae65bd,110fcb6b6a726619] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/47/logSegments client=10.11.12.102, traceId=e00faddbeeae65bd, duration=5 ms] 2025-04-08 20:47:23.105 INFO 3009027 --- [ocp-schedules-27,dfd29da0511f046a,a00b980cf34a9be6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 20:47:23.131 INFO 3009027 --- [ocp-schedules-2,8871eec4f6d4867a,4989264ef6ec4bbb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:47:23.137 INFO 3009027 --- [ocp-schedules-19,8a18d2b5c0c26222,7462925eab47f49c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:47:23.137 INFO 3009027 --- [ocp-schedules-19,8a18d2b5c0c26222,7462925eab47f49c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:47:23.137 INFO 3009027 --- [ocp-schedules-19,8a18d2b5c0c26222,7462925eab47f49c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:47:23.137 INFO 3009027 --- [ocp-schedules-19,8a18d2b5c0c26222,7462925eab47f49c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:47:23.138 INFO 3009027 --- [ocp-schedules-19,8a18d2b5c0c26222,7462925eab47f49c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:47:23.138 INFO 3009027 --- [ocp-schedules-19,8a18d2b5c0c26222,7462925eab47f49c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:47:23.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:47:23.175979+08:00, lastProcessTime=2025-04-08T20:47:13.175616+08:00 2025-04-08 20:47:23.213 INFO 3009027 --- [ocp-schedules-20,cef4de007efffe34,c27b135279d0ad1c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:47:23.214 INFO 3009027 --- [ocp-schedules-20,cef4de007efffe34,c27b135279d0ad1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:23.230 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:47:23.232 ERROR 3009027 --- [ocp-schedules-20,cef4de007efffe34,c27b135279d0ad1c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:47:23.232 INFO 3009027 --- [ocp-schedules-20,cef4de007efffe34,c27b135279d0ad1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:47:23.266 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:47:23.175979+08:00 2025-04-08 20:47:23.331 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:23.331 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:23.990 INFO 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:47:23.990 INFO 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:47:23.992 INFO 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:47:23.993 WARN 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:47:23.993 INFO 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:47:23.994 WARN 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:47:23.999 INFO 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:47:24.002 WARN 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:47:24.002 ERROR 3009027 --- [ocp-schedules-8,9d924b72f067c950,605eb85af20f03db] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:47:24.306 INFO 3009027 --- [ocp-schedules-17,412f636286bab2fe,6553ce553da0b7f1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:47:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,eec7c99165b29d9b,cff6f0344bf0e97c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:25.019 WARN 3009027 --- [ocp-async-6,461548855a94216d,3f937927ed0b92cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:25.019 INFO 3009027 --- [ocp-async-7,8f78624081096c94,6bb9a25e66603269] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:25.777 INFO 3009027 --- [ocp-schedules-27,da81c48a802f90bd,ef4c0febfede55b3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:47:25.874 INFO 3009027 --- [ocp-schedules-27,da81c48a802f90bd,ef4c0febfede55b3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:47:25.874 INFO 3009027 --- [ocp-schedules-27,da81c48a802f90bd,ef4c0febfede55b3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:47:26.332 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:26.332 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:47:27.019 WARN 3009027 --- [ocp-async-1,d5f3c743ce420e5b,04599cddd901cc56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:27.019 INFO 3009027 --- [ocp-async-2,c6c00ac05af0f087,f8102e01bbf21d67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,43e0b434c8fbb2a1,a03b7434e0aab4db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:27.454 INFO 3009027 --- [ocp-schedules-21,cbae694ead1cd29b,7521a89a0fc11f34] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:47:29.019 WARN 3009027 --- [ocp-async-9,34bed6dbf5613856,9dfd9f6664013d26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:29.019 INFO 3009027 --- [ocp-async-10,77849c5b55f05d9c,525ec2e3d3974d1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:29.320 INFO 3009027 --- [ocp-schedules-19,da155712fa404ff8,b877417bd0b1d9ef] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:47:29.321 INFO 3009027 --- [ocp-schedules-19,da155712fa404ff8,b877417bd0b1d9ef] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:47:29.332 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:29.333 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,c8b83d0d250ff779,16fac82e729a7797] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:31.019 WARN 3009027 --- [ocp-async-8,c4d1c2ac5a503e4c,5535be82e7897758] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:31.019 INFO 3009027 --- [ocp-async-3,8db355aee71a98ad,6432ab444004e41a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:32.333 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:32.334 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:32.897 INFO 3009027 --- [ocp-schedules-12,2c038747bbb2087a,0605c9ea08142c40] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:47:32.898 INFO 3009027 --- [ocp-schedules-15,98c1193226fdebec,ddca928be5ef38b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:47:32.950 INFO 3009027 --- [ocp-async-7,39cf4d811b93a30c,55c4cf69a063a19e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:47:33.019 WARN 3009027 --- [ocp-async-9,1438de3f42ea07e0,0dfd1f49a0eb0052] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:33.019 INFO 3009027 --- [ocp-async-2,d12d6c24c1f6acd1,648d540d8afe1c01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:33.058 INFO 3009027 --- [ocp-schedules-2,a75d763227a5b788,dd78d51252f9bb7d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:47:33.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:47:33.177177+08:00, lastProcessTime=2025-04-08T20:47:23.175979+08:00 2025-04-08 20:47:33.275 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:47:33.177177+08:00 2025-04-08 20:47:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,470b3d9dca195eff,200e2fe973c7320c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:35.019 WARN 3009027 --- [ocp-async-10,ac3afde4ad3f5535,6c6fe5106dc817c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:35.019 INFO 3009027 --- [ocp-async-8,92ddbe0a75d8e082,12e8163818322efd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:35.334 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:35.334 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,824c2c53c071d720,841c606228f1be2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:37.019 WARN 3009027 --- [ocp-async-8,9e13aef2e3cb2643,cb0d8dd9b9c2ff86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:37.019 INFO 3009027 --- [ocp-async-3,5e4da10b3774eaf6,2600cf78e1927ede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:37.483 INFO 3009027 --- [ocp-schedules-7,6065bb7be07a042c,9bb997380838951a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:47:38.335 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:38.335 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:39.019 WARN 3009027 --- [ocp-async-3,bab21387a054f68f,92c61841accdd26a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:39.020 INFO 3009027 --- [ocp-async-1,b566046c90c0bdf3,2f7116d6eddfc964] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:39.331 INFO 3009027 --- [ocp-schedules-28,72a689a6e4066fb1,7518586be7b88ed3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:47:39.332 INFO 3009027 --- [ocp-schedules-28,72a689a6e4066fb1,7518586be7b88ed3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:47:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,42aa0878737ac167,42d9f4114f45cf33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:40.001 INFO 3009027 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:47:40.001 INFO 3009027 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:47:41.020 WARN 3009027 --- [ocp-async-10,9bf73740f119a56a,96844975d857e9e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:41.020 INFO 3009027 --- [ocp-async-2,8db800235e087fed,6abd617a0ccbe313] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:41.335 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:41.336 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,b41dee1b4490bbae,6dba4c729e379cb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:42.951 INFO 3009027 --- [ocp-async-2,5e5aa8cff4129a31,192de679778598f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:47:42.952 INFO 3009027 --- [ocp-schedules-9,cfc6f425c87a40e3,d8b0cc0596a16b37] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:47:42.953 INFO 3009027 --- [ocp-schedules-9,cfc6f425c87a40e3,d8b0cc0596a16b37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:42.953 INFO 3009027 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:47:43.020 WARN 3009027 --- [ocp-async-7,3595fcf84c402de2,59594dd9fbe0830d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:43.020 INFO 3009027 --- [ocp-async-8,ecad17d7989eac08,875e6b0485f3437c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:43.059 INFO 3009027 --- [ocp-schedules-18,5fa15ee79fc08399,25e5bbcba538f1de] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:47:43.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:47:43.177072+08:00, lastProcessTime=2025-04-08T20:47:33.177177+08:00 2025-04-08 20:47:43.288 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:47:43.177072+08:00 2025-04-08 20:47:44.336 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:44.336 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:45.020 WARN 3009027 --- [ocp-async-8,f7b199ae005f3800,532516251bb8b363] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:45.020 INFO 3009027 --- [ocp-async-9,299f3d8beea96558,e25d69b441305312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:45.408 INFO 3009027 --- [prometheus-minute-schedule-1,b3d1c87c2f2b08ca,8b86b0ad5a97fe34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:45.875 INFO 3009027 --- [ocp-schedules-11,e80602dbafd31359,394a5ee1f81f617b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:47:45.881 INFO 3009027 --- [ocp-schedules-11,e80602dbafd31359,394a5ee1f81f617b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:47:45.881 INFO 3009027 --- [ocp-schedules-11,e80602dbafd31359,394a5ee1f81f617b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:47:47.020 WARN 3009027 --- [ocp-async-10,82a4da8a4d5645ac,2df21593978ab4fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:47.020 INFO 3009027 --- [ocp-async-1,163c5f9e8a111861,a5bdc7ef1d279050] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:47.337 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:47.337 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:47.504 INFO 3009027 --- [ocp-schedules-2,0dfabaa978dc3b66,8d58eaaf1e953899] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:47:48.408 INFO 3009027 --- [prometheus-minute-schedule-1,4da682b77aa872f3,a1a1d24a8cfe0daa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:49.020 WARN 3009027 --- [ocp-async-7,b8d98c53409926d0,27bc2c8d1a5178a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:49.020 INFO 3009027 --- [ocp-async-6,4d5b4351143f0804,68ce90587b1c75a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:49.336 INFO 3009027 --- [ocp-schedules-21,65200e131297cab8,1e1b31da6e351a3b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:47:49.337 INFO 3009027 --- [ocp-schedules-21,65200e131297cab8,1e1b31da6e351a3b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:47:50.337 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:50.338 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:51.020 WARN 3009027 --- [ocp-async-8,f7438bad8e915255,71f71abfd9808df5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:51.020 INFO 3009027 --- [ocp-async-4,a5e2566d10a347fc,05c15a10ded66164] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:51.408 INFO 3009027 --- [prometheus-minute-schedule-1,14393d1128882645,e0dc9654d8fdbb90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:52.896 INFO 3009027 --- [ocp-schedules-6,8b9cb6a18bfe18e0,96695f2e6b371248] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:47:52.896 INFO 3009027 --- [ocp-schedules-8,75590dce0a8d90fc,faf227d0c95e11aa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:47:52.896 INFO 3009027 --- [ocp-schedules-8,75590dce0a8d90fc,faf227d0c95e11aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:52.896 INFO 3009027 --- [ocp-schedules-18,bd2931b977b5e554,18607f2954de9883] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:47:52.897 INFO 3009027 --- [ocp-schedules-18,bd2931b977b5e554,18607f2954de9883] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:52.897 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:47:52.898 INFO 3009027 --- [ocp-schedules-14,b991070f9347e885,2128bf6e1af269c9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:47:52.898 INFO 3009027 --- [ocp-schedules-26,0fa02666101e562b,9d846d6547309653] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:47:52.898 INFO 3009027 --- [ocp-schedules-14,b991070f9347e885,2128bf6e1af269c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:47:52.898 INFO 3009027 --- [ocp-schedules-2,3170f23c848790ca,979d65a895c2266a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:47:52.898 INFO 3009027 --- [ocp-schedules-26,0fa02666101e562b,9d846d6547309653] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:47:52.898 INFO 3009027 --- [ocp-schedules-2,3170f23c848790ca,979d65a895c2266a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:47:52.904 INFO 3009027 --- [ocp-schedules-29,8c5a5c7ac288d5a4,cdd7e0603064c739] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:47:52.951 INFO 3009027 --- [ocp-async-4,a838d3ef8a3fdd43,8451b08bfc55bcd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:47:52.953 INFO 3009027 --- [ocp-schedules-9,cfc6f425c87a40e3,d8b0cc0596a16b37] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:47:52.968 INFO 3009027 --- [ocp-schedules-4,bdcbf535eec50fd0,fd0d230054ad6cc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:47:52.968 INFO 3009027 --- [ocp-schedules-4,e8255c575d3c78de,b96101e55adbdfe5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:47:52.969 INFO 3009027 --- [ocp-schedules-4,e8255c575d3c78de,b96101e55adbdfe5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:47:52.974 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:47:52.974 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:47:53.002 INFO 3009027 --- [ocp-schedules-19,52b449f056b56dd2,c2d84ea0a70fed54] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:47:53.021 WARN 3009027 --- [ocp-async-2,a47cadfae9e6af35,923b9bf8f789d558] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:53.021 INFO 3009027 --- [ocp-async-5,742d6a6e0aa72bd6,b082aa416af237c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:53.061 INFO 3009027 --- [ocp-schedules-23,36b8901c278bcddb,46ff9c9551d07a46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:47:53.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:47:53.177544+08:00, lastProcessTime=2025-04-08T20:47:43.177072+08:00 2025-04-08 20:47:53.291 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:47:53.177544+08:00 2025-04-08 20:47:53.338 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:53.339 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:54.408 INFO 3009027 --- [prometheus-minute-schedule-1,7b2427bbb1935244,6a5560e5516d4ac1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:55.021 WARN 3009027 --- [ocp-async-6,637a419e6561f210,1b704cacf506efa5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:55.021 INFO 3009027 --- [ocp-async-1,41c45bca867af846,affd31cecf9f8011] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:56.339 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:56.339 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:57.021 INFO 3009027 --- [ocp-async-3,d895eaaf3d39e1ef,970b86f021155081] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:57.021 WARN 3009027 --- [ocp-async-9,652541e755f3e7b6,d1391f9298d53ba0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:57.408 INFO 3009027 --- [prometheus-minute-schedule-1,8818feeda14c7f18,fb39c9de65311cb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116480, interval=60 2025-04-08 20:47:57.531 INFO 3009027 --- [ocp-schedules-12,08b5a8e971de6774,d67e2898704363a0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:47:59.021 WARN 3009027 --- [ocp-async-4,b21e57b6a0e7f5d3,4844e6b078c6bfa9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:47:59.021 INFO 3009027 --- [ocp-async-10,9dba24dac2f5f7f4,bea3ee2b462ffcfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:47:59.339 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:47:59.340 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:47:59.342 INFO 3009027 --- [ocp-schedules-5,ffbb0434c9fef567,a40313861f7d4ca2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:47:59.343 INFO 3009027 --- [ocp-schedules-5,ffbb0434c9fef567,a40313861f7d4ca2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:48:00.408 INFO 3009027 --- [prometheus-minute-schedule-1,382d8267c7253307,541c56d2aa2928c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:01.021 WARN 3009027 --- [ocp-async-7,9df692404c43b852,ceca2c713257418b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:01.021 INFO 3009027 --- [ocp-async-6,0df07bab1a4c0c5a,cdc204c10de75e1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:02.340 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:02.340 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:02.897 INFO 3009027 --- [ocp-schedules-8,75590dce0a8d90fc,faf227d0c95e11aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:48:02.898 INFO 3009027 --- [ocp-schedules-18,bd2931b977b5e554,18607f2954de9883] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:48:02.951 INFO 3009027 --- [ocp-async-6,95832792ada452bf,678e26f740754dd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:48:03.021 INFO 3009027 --- [ocp-async-9,d9e430c0ebebab4c,468f0da20aaed24e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:03.021 WARN 3009027 --- [ocp-async-1,970fee3fc50327d7,1baa56ffc4824e34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:03.062 INFO 3009027 --- [ocp-schedules-7,16ef594f6d2ecf35,b60b83388b3e1fc5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:48:03.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:48:03.177789+08:00, lastProcessTime=2025-04-08T20:47:53.177544+08:00 2025-04-08 20:48:03.256 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,750e9453576893c3,84ee0603756ad63e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/73/logSegments?taskInstanceId=8&subtaskInstanceId=73, client=10.11.12.102, traceId=750e9453576893c3, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,73,] 2025-04-08 20:48:03.258 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-9,750e9453576893c3,84ee0603756ad63e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/73/logSegments client=10.11.12.102, traceId=750e9453576893c3, duration=5 ms] 2025-04-08 20:48:03.270 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:48:03.177789+08:00 2025-04-08 20:48:03.408 INFO 3009027 --- [prometheus-minute-schedule-1,849553b9948b74d5,e1eca6f6e7549ec4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:05.022 WARN 3009027 --- [ocp-async-10,75287e90a07bb028,7463e8a03c0d69b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:05.022 INFO 3009027 --- [ocp-async-4,ded780ed0d96b689,6582787baf23592c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:05.341 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:05.341 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:05.882 INFO 3009027 --- [ocp-schedules-18,b502da9555ee564d,6329f763eb992407] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:48:05.937 INFO 3009027 --- [ocp-schedules-18,b502da9555ee564d,6329f763eb992407] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:48:05.937 INFO 3009027 --- [ocp-schedules-18,b502da9555ee564d,6329f763eb992407] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:48:06.408 INFO 3009027 --- [prometheus-minute-schedule-1,5a1c238bec4e3f76,5da6fe0ede2b69c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:07.022 WARN 3009027 --- [ocp-async-5,4edf1a3ea84971ea,14a589caa68db02a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:07.022 INFO 3009027 --- [ocp-async-4,bceaf119c2d4785d,077d2a1752b9f426] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:07.553 INFO 3009027 --- [ocp-schedules-17,648f4894202b541d,1ca294e165a7bce1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:48:08.341 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:08.342 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:09.022 WARN 3009027 --- [ocp-async-4,13f508d714133165,1e3fa6bbf508e5c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:09.022 INFO 3009027 --- [ocp-async-3,c7a6e16371406662,bec418bc335cef83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:09.349 INFO 3009027 --- [ocp-schedules-20,e7ce6eee11e4242b,1fe79293805aed15] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:48:09.350 INFO 3009027 --- [ocp-schedules-20,e7ce6eee11e4242b,1fe79293805aed15] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:48:09.408 INFO 3009027 --- [prometheus-minute-schedule-1,86acf645ac7c4e8b,9f27c227efca5ee9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:10.001 INFO 3009027 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:48:10.002 INFO 3009027 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:48:10.988 INFO 3009027 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 20:48:11.022 WARN 3009027 --- [ocp-async-10,c7f000415dd3cc76,7b9014a1209a2694] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:11.022 INFO 3009027 --- [ocp-async-2,5f5a7c8a2200a370,9894728e10f205f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:11.342 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:11.342 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:12.408 INFO 3009027 --- [prometheus-minute-schedule-1,5a7aaf1ae7cb641d,c2ac66b7aeda76aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:12.408 INFO 3009027 --- [prometheus-cache-monitor1,084c8b6c2b511d5e,a22b63e43aa3e19c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":423,"secondWriteQueue":422,"minuteWriteCacheSize":420} 2025-04-08 20:48:12.408 INFO 3009027 --- [prometheus-cache-monitor1,084c8b6c2b511d5e,a22b63e43aa3e19c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:48:12.507 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,103f167bd80efd90,010ef6b387f6c926] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/58/logSegments?taskInstanceId=8&subtaskInstanceId=58, client=10.11.12.102, traceId=103f167bd80efd90, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,58,] 2025-04-08 20:48:12.510 INFO 3009027 --- [http-nio-0.0.0.0-8080-exec-6,103f167bd80efd90,010ef6b387f6c926] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/58/logSegments client=10.11.12.102, traceId=103f167bd80efd90, duration=6 ms] 2025-04-08 20:48:12.730 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744116420, nodeCount=422, affectRows=422 2025-04-08 20:48:12.733 INFO 3009027 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744116420, nodeCount=419, validCount=419 2025-04-08 20:48:12.951 INFO 3009027 --- [ocp-async-2,a4a75f3379ffaa53,ad24f2eb19e34c7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:48:13.022 WARN 3009027 --- [ocp-async-8,6cc634a9b1bbcae2,9dbd6c1cf41a1705] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:13.022 INFO 3009027 --- [ocp-async-1,9382d7fa2980ceda,4cff42040ac188de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:13.064 INFO 3009027 --- [ocp-schedules-3,a84f5c346d4bf5d1,46adbe1c96f8e541] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:48:13.176 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:48:13.175600+08:00, lastProcessTime=2025-04-08T20:48:03.177789+08:00 2025-04-08 20:48:13.266 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:48:13.175600+08:00 2025-04-08 20:48:14.343 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:14.343 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:15.022 WARN 3009027 --- [ocp-async-6,bfc99dc4e525cb65,c75f06c6f20d0671] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:15.022 INFO 3009027 --- [ocp-async-4,943e61781d37e191,2d0f7d03c9400702] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:15.049 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:34:44+08:00, updateTime=2025-04-08T20:47:44+08:00, expireTime=null, isLeader=false} 2025-04-08 20:48:15.050 WARN 3009027 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:48:15.408 INFO 3009027 --- [prometheus-minute-schedule-1,7da366f3363af7f8,54aaa9eaafe4b371] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:17.022 WARN 3009027 --- [ocp-async-10,ac50daf7dd4fbfda,863d61d857be7c87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:17.022 INFO 3009027 --- [ocp-async-2,443fb1dee4430038,f9c23176b688d15c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:17.344 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:17.344 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:17.581 INFO 3009027 --- [ocp-schedules-17,2cde8f7c04c11153,0f843826ece9af7b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:48:18.408 INFO 3009027 --- [prometheus-minute-schedule-1,a0ad1060c31cc615,89c44f83a641582f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:19.023 WARN 3009027 --- [ocp-async-7,a6af12af7f76c073,a5bd747ed37faaac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:19.023 INFO 3009027 --- [ocp-async-5,23a5c98400939dbe,3fb79c50b29dc6d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:19.353 INFO 3009027 --- [ocp-schedules-9,d014a90394e8a8ea,21270fd937cce033] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:48:19.354 INFO 3009027 --- [ocp-schedules-9,d014a90394e8a8ea,21270fd937cce033] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:48:20.344 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:20.345 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:21.023 WARN 3009027 --- [ocp-async-6,1e349ff5e03a0f5f,3e92269992b0453e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:21.023 INFO 3009027 --- [ocp-async-3,ab577b07f45bc714,7d53b5b8873112d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:21.408 INFO 3009027 --- [prometheus-minute-schedule-1,747c714b1525b999,4c5521562dbaaeac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:22.895 INFO 3009027 --- [ocp-schedules-16,6b78063de468876e,262cff8229b2a68c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:48:22.896 INFO 3009027 --- [ocp-schedules-13,db14365a79f97354,63bee67508dac3c3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:48:22.896 INFO 3009027 --- [ocp-schedules-10,8a6ab5cf9138acd6,bf115b4e22733861] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:48:22.896 INFO 3009027 --- [ocp-schedules-11,a0150e61c9dbf5f9,f4a059f0cecd557d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:48:22.896 INFO 3009027 --- [ocp-schedules-10,8a6ab5cf9138acd6,bf115b4e22733861] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:48:22.896 INFO 3009027 --- [ocp-schedules-15,9397e7121358926c,fe60d735322fce30] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-15,9397e7121358926c,fe60d735322fce30] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-9,6e8c899eb396070a,8026931dc527c835] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 20:48:22.898 INFO 3009027 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-9,6e8c899eb396070a,8026931dc527c835] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-6,245bbb1dc482f7df,645fc17ff3a6efb0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-6,245bbb1dc482f7df,645fc17ff3a6efb0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-2,acb0f4f9221d9f77,592af475df798e3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 20:48:22.898 INFO 3009027 --- [ocp-schedules-2,acb0f4f9221d9f77,592af475df798e3b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 20:48:22.903 INFO 3009027 --- [ocp-schedules-11,a0150e61c9dbf5f9,f4a059f0cecd557d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:48:22.912 INFO 3009027 --- [ocp-schedules-16,6b78063de468876e,262cff8229b2a68c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:48:22.914 INFO 3009027 --- [ocp-schedules-3,95798b146cad7d9b,24fa98842c77a709] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:48:22.915 INFO 3009027 --- [ocp-schedules-14,ae0b543a96cbc352,15b07b9fb8d60a25] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1688,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":420,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":420} 2025-04-08 20:48:22.915 INFO 3009027 --- [ocp-schedules-14,ae0b543a96cbc352,15b07b9fb8d60a25] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 20:48:22.951 INFO 3009027 --- [ocp-async-10,2da3f5e51dd2b2de,6346cbfb2c9dfa7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:48:22.971 INFO 3009027 --- [ocp-schedules-27,5c065446d6e910f2,b221bea82d702516] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:48:22.974 INFO 3009027 --- [ocp-schedules-4,e8255c575d3c78de,b96101e55adbdfe5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:48:22.974 INFO 3009027 --- [ocp-schedules-4,553919d0953aa21b,67c7cce81dd58859] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:48:22.975 INFO 3009027 --- [ocp-schedules-4,553919d0953aa21b,67c7cce81dd58859] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:48:22.979 INFO 3009027 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 20:48:22.980 INFO 3009027 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:48:23.003 INFO 3009027 --- [ocp-schedules-30,096ab7af0dbfab42,1afe5a9309c73909] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:48:23.003 INFO 3009027 --- [ocp-schedules-5,a3e1dbd4ef942ce5,44944179c04e6f51] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:48:23.003+08:00, clusters=[2] 2025-04-08 20:48:23.004 INFO 3009027 --- [ocp-schedules-5,a3e1dbd4ef942ce5,44944179c04e6f51] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:48:23.009 INFO 3009027 --- [ocp-schedules-28,0996317473989f24,d8c9471d4210d450] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:48:23.009 INFO 3009027 --- [ocp-schedules-28,0996317473989f24,d8c9471d4210d450] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:48:23.023 WARN 3009027 --- [ocp-async-2,6ddad16b7987250e,f59b8377fcec81af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:23.023 INFO 3009027 --- [ocp-async-7,7bece37a2a679f74,ee4e9a0270340409] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:23.025 INFO 3009027 --- [ocp-schedules-13,fee11d0ea1e6e942,2286d99bf8ec6188] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:48:23.025 INFO 3009027 --- [ocp-schedules-13,fee11d0ea1e6e942,2286d99bf8ec6188] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:48:23.028 INFO 3009027 --- [ocp-schedules-11,5bf0dd2666ea73a7,5f672eff23972d17] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:48:23.029 INFO 3009027 --- [ocp-schedules-11,5bf0dd2666ea73a7,5f672eff23972d17] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:48:23.029 INFO 3009027 --- [ocp-schedules-11,5bf0dd2666ea73a7,5f672eff23972d17] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:48:23.032 INFO 3009027 --- [ocp-schedules-7,f020562ec856e0a4,3c6625522ee77912] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:48:23.032+08:00, clusters=[2] 2025-04-08 20:48:23.034 INFO 3009027 --- [ocp-schedules-7,f020562ec856e0a4,3c6625522ee77912] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:48:23.044 INFO 3009027 --- [ocp-schedules-3,9bf1d5a34d780b33,0f028ef9851bc2b7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:48:23.044 INFO 3009027 --- [ocp-schedules-3,9bf1d5a34d780b33,0f028ef9851bc2b7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:48:23.044 INFO 3009027 --- [ocp-schedules-21,7f3ec96a68bfb1d0,aab8377f3203e00a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:48:23.045 INFO 3009027 --- [ocp-schedules-21,7f3ec96a68bfb1d0,aab8377f3203e00a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:48:23.048 INFO 3009027 --- [ocp-schedules-3,9bf1d5a34d780b33,0f028ef9851bc2b7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:48:23.048 INFO 3009027 --- [ocp-schedules-3,9bf1d5a34d780b33,0f028ef9851bc2b7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:48:23.055 ERROR 3009027 --- [ocp-schedules-21,7f3ec96a68bfb1d0,aab8377f3203e00a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$8eed97a8.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:48:23.056 INFO 3009027 --- [ocp-schedules-21,7f3ec96a68bfb1d0,aab8377f3203e00a] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:48:23.065 INFO 3009027 --- [ocp-schedules-16,47deb1b5f4217950,d3e44cdb53d91f52] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:48:23.106 INFO 3009027 --- [ocp-schedules-1,e039c15a5dd28553,61845c4c8534d8e8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:48:23.107 INFO 3009027 --- [ocp-schedules-1,e039c15a5dd28553,61845c4c8534d8e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:48:23.118 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:48:23.118 INFO 3009027 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 20:48:23.137 INFO 3009027 --- [ocp-schedules-26,e37408397495607e,cef57a64fcbe4956] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:48:23.138 INFO 3009027 --- [ocp-schedules-23,4e8893530a2957a5,27040890042b1a1e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:48:23.138 INFO 3009027 --- [ocp-schedules-23,4e8893530a2957a5,27040890042b1a1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:48:23.138 INFO 3009027 --- [ocp-schedules-23,4e8893530a2957a5,27040890042b1a1e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:48:23.138 INFO 3009027 --- [ocp-schedules-23,4e8893530a2957a5,27040890042b1a1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 20:48:23.139 INFO 3009027 --- [ocp-schedules-23,4e8893530a2957a5,27040890042b1a1e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:48:23.139 INFO 3009027 --- [ocp-schedules-23,4e8893530a2957a5,27040890042b1a1e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 20:48:23.177 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:48:23.176590+08:00, lastProcessTime=2025-04-08T20:48:13.175600+08:00 2025-04-08 20:48:23.232 INFO 3009027 --- [ocp-schedules-12,27e9bc2daba5f415,772c5a35404c6d54] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:48:23.233 INFO 3009027 --- [ocp-schedules-12,27e9bc2daba5f415,772c5a35404c6d54] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:48:23.248 ERROR 3009027 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 20:48:23.249 ERROR 3009027 --- [ocp-schedules-12,27e9bc2daba5f415,772c5a35404c6d54] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor843.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor401.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$1d4ed25d.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:48:23.249 INFO 3009027 --- [ocp-schedules-12,27e9bc2daba5f415,772c5a35404c6d54] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 20:48:23.269 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:48:23.176590+08:00 2025-04-08 20:48:23.345 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:23.346 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:24.006 INFO 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 20:48:24.006 INFO 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-04-08 20:48:24.008 INFO 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:48:24.009 WARN 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:48:24.010 INFO 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:48:24.010 WARN 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:48:24.014 INFO 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 20:48:24.016 WARN 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 20:48:24.016 ERROR 3009027 --- [ocp-schedules-11,69b75dee27be773f,9747f43d8b816e8d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor910.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$2e55aed0.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 20:48:24.312 INFO 3009027 --- [ocp-schedules-16,63b0238656a8af8e,145d6248a9ae1554] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:48:24.408 INFO 3009027 --- [prometheus-minute-schedule-1,bee1e521f67a3600,3d8795acd4c56939] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:25.023 WARN 3009027 --- [ocp-async-6,c2b0a628ec522c79,628932f6640dacab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:25.023 INFO 3009027 --- [ocp-async-7,e921ec4e9fd8a6c2,02f20e4e00f8254d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:25.939 INFO 3009027 --- [ocp-schedules-29,d75514d20ba26de0,f36b731829a3113c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 20:48:25.954 INFO 3009027 --- [ocp-schedules-29,d75514d20ba26de0,f36b731829a3113c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:48:25.954 INFO 3009027 --- [ocp-schedules-29,d75514d20ba26de0,f36b731829a3113c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 20:48:26.346 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:26.346 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:26.977 INFO 3009027 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 20:48:27.023 WARN 3009027 --- [ocp-async-5,20bd03257d57e0eb,8cf32dacfbfb7028] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:27.023 INFO 3009027 --- [ocp-async-3,e8bcc02ed5e78f9c,594ce29a36951b43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:27.408 INFO 3009027 --- [prometheus-minute-schedule-1,4d149ef4b7951149,7c8be1a4d222caaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:27.601 INFO 3009027 --- [ocp-schedules-27,5450b1a7d230c2e2,ce287d898bbcf0f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:48:29.023 WARN 3009027 --- [ocp-async-2,b98310fde72631d2,070aefa112931be7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:29.023 INFO 3009027 --- [ocp-async-8,4ffeeb1812ae2ab7,1af5b4f966aec2b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:29.347 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:29.347 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:29.358 INFO 3009027 --- [ocp-schedules-7,51e38d9ee0d8c970,ad413ad784e61794] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:48:29.360 INFO 3009027 --- [ocp-schedules-7,51e38d9ee0d8c970,ad413ad784e61794] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:48:30.408 INFO 3009027 --- [prometheus-minute-schedule-1,d9c1411094a16a68,b856e3be05f025af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:31.024 WARN 3009027 --- [ocp-async-4,4411853de8080814,ffe9116d8bd21d02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:31.024 INFO 3009027 --- [ocp-async-1,5675714629e68d87,ec2e95ed8bb840dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:32.347 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:32.348 INFO 3009027 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:32.857 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:48:32.857 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:48:32.860 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:48:32.860 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:48:32.860 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:48:32.860 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:48:32.863 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:48:32.863 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:48:32.863 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:48:32.863 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:48:32.885 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:48:32.890 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:48:32.890 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:48:32.890 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:48:32.890 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:48:32.891 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:48:32.896 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:48:32.897 INFO 3009027 --- [ocp-schedules-10,8a6ab5cf9138acd6,bf115b4e22733861] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 20:48:32.898 INFO 3009027 --- [ocp-schedules-15,9397e7121358926c,fe60d735322fce30] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 20:48:32.951 INFO 3009027 --- [ocp-async-7,4f1cfc9a686d96a3,08dbddeedc42a2c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:48:33.024 WARN 3009027 --- [ocp-async-3,73e2f420154ea011,9770e7dd111e7986] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:33.024 INFO 3009027 --- [ocp-async-9,e631e5f8037938a2,f683bb2eb9c011d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:33.066 INFO 3009027 --- [ocp-schedules-9,d0dc320d9dc2e0bf,0175d627fe7e1eaa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:48:33.178 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T20:48:33.176923+08:00, lastProcessTime=2025-04-08T20:48:23.176590+08:00 2025-04-08 20:48:33.203 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:48:33.264 INFO 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T20:48:33.176923+08:00 2025-04-08 20:48:33.408 INFO 3009027 --- [prometheus-minute-schedule-1,54547abe064411c1,9fb8dd2051d24c1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:33.514 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.650866509S 2025-04-08 20:48:33.514 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:48:33.570 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 20:48:33.570 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 20:48:33.595 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 20:48:33.634 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:48:33.634 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:48:33.634 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:48:33.634 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:48:33.634 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:48:33.634 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 20:48:33.641 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:48:33.746 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] o.s.boot.SpringApplication : Started application in 0.886 seconds (JVM running for 1555.363) 2025-04-08 20:48:33.746 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.886683452S 2025-04-08 20:48:34.018 INFO 3009027 --- [ocp-schedules-12,3c1655948f76a1aa,3819044fd1030011] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 20:48:35.024 WARN 3009027 --- [ocp-async-10,a0ddf479573f7d2d,77fdd376aa93be78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:35.024 INFO 3009027 --- [ocp-async-9,13145f85b32e6a5c,098d325317c5d9aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:35.348 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:35.349 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:36.408 INFO 3009027 --- [prometheus-minute-schedule-1,bde5233cbe7ec7f8,889deddd5d79f817] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:37.024 WARN 3009027 --- [ocp-async-9,8d3f9ad9f7b2b4fa,cdd1eaeed6a9cdea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:37.024 INFO 3009027 --- [ocp-async-1,82f83b09435a709f,197d27af28744f91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:37.626 INFO 3009027 --- [ocp-schedules-27,1afdff1e1ffb45de,d57c5f5c063eb9c5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 20:48:38.349 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:38.349 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:39.024 WARN 3009027 --- [ocp-async-3,277602c29f63c243,2a29b26a3961066e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:39.024 INFO 3009027 --- [ocp-async-2,672488c7ecf3a65f,f30d59b04fffe1de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:39.362 INFO 3009027 --- [ocp-schedules-8,1a8007219571a619,7231cf2d129178ea] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:48:39.364 INFO 3009027 --- [ocp-schedules-8,1a8007219571a619,7231cf2d129178ea] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:48:39.408 INFO 3009027 --- [prometheus-minute-schedule-1,3f786e8e76987f11,e4ac798fd7ebc85c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:40.001 INFO 3009027 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 20:48:40.002 INFO 3009027 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 20:48:40.968 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:48:40.969 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:48:40.969 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:48:40.969 INFO 3009027 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 20:48:40.970 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 20:48:40.974 WARN 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(2), status=TAKINGOVER 2025-04-08 20:48:40.974 INFO 3009027 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 20:48:41.024 WARN 3009027 --- [ocp-async-2,fe1f09edcc691d48,254a151ca213a358] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:41.024 INFO 3009027 --- [ocp-async-10,b5052f45c74896a6,1046d8c2ed582dfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:41.349 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:48:41.350 INFO 3009027 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:48:42.408 INFO 3009027 --- [prometheus-minute-schedule-1,9af41d9487a66138,40e624e75740fe12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744116540, interval=60 2025-04-08 20:48:42.952 INFO 3009027 --- [ocp-async-8,a475335cf760d63a,d78c3bd23772d29c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:48:43.024 WARN 3009027 --- [ocp-async-9,21d612e15b67b4f6,ca0a89b54a75952d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:48:43.024 INFO 3009027 --- [ocp-async-6,9a348b06e60b4bed,b8271e111517311b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:48:43.068 INFO 3009027 --- [ocp-schedules-10,40835f29aa2a0f51,6bc833fbb2cc0612] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:48:43.138 ERROR 3009027 --- [sshj-Reader-/192.168.1.61:22,,] n.schmizz.sshj.transport.TransportImpl : Dying because - Broken transport; encountered EOF net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF at net.schmizz.sshj.transport.Reader.run(Reader.java:58) 2025-04-08 20:48:43.138 INFO 3009027 --- [sshj-Reader-/192.168.1.61:22,,] n.schmizz.sshj.transport.TransportImpl : Disconnected - UNKNOWN 2025-04-08 20:48:43.141 WARN 3009027 --- [Thread-50,,] c.o.o.e.internal.connector.Connectors : [Connectors]: shutdown executor service. 2025-04-08 20:48:43.141 WARN 3009027 --- [Thread-50,,] c.o.o.e.internal.connector.Connectors : [Connectors]: shutdown connection manager. 2025-04-08 20:48:43.211 WARN 3009027 --- [alarm-task-1,84a40531e0a4abb4,3f31f46e878ce67f] com.zaxxer.hikari.pool.ProxyConnection : metadb-connect-pool - Connection com.oceanbase.jdbc.OceanBaseConnection@5023edea marked as broken because of SQLSTATE(08000), ErrorCode(0) java.sql.SQLNonTransientConnectionException: (conn=3221489516) Connection reset at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:122) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseConnection.setReadOnly(OceanBaseConnection.java:1252) at com.zaxxer.hikari.pool.ProxyConnection.setReadOnly(ProxyConnection.java:423) at com.zaxxer.hikari.pool.HikariProxyConnection.setReadOnly(HikariProxyConnection.java) at org.springframework.jdbc.datasource.DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:190) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:153) at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421) at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at com.sun.proxy.$Proxy380.findAll(Unknown Source) at com.oceanbase.ocp.alarm.service.OcpAlarmChannelService.listAll(OcpAlarmChannelService.java:355) at com.oceanbase.ocp.alarm.service.OcpAlarmChannelService$$FastClassBySpringCGLIB$$e4f7fa3b.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.alarm.service.OcpAlarmChannelService$$EnhancerBySpringCGLIB$$98377f9f.listAll() at com.oceanbase.ocp.alarm.core.aggregate.AggregatorFactory.refresh(AggregatorFactory.java:95) at com.oceanbase.ocp.alarm.core.process.AlarmProcessor.process(AlarmProcessor.java:105) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.alarm.core.task.AlarmProcessTask.processOnce(AlarmProcessTask.java:40) at com.oceanbase.ocp.alarm.utils.task.AbstractIntervalTask.call(AbstractIntervalTask.java:41) at com.oceanbase.ocp.alarm.utils.task.AbstractIntervalTask.call(AbstractIntervalTask.java:15) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Connection reset at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:329) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:298) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.setReadonly(AbstractQueryProtocol.java:2057) at com.oceanbase.jdbc.OceanBaseConnection.setReadOnly(OceanBaseConnection.java:1249) ... 37 common frames omitted Caused by: java.sql.SQLNonTransientConnectionException: Connection reset at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2730) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2213) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:324) ... 40 common frames omitted Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.FilterInputStream.read(FilterInputStream.java:133) at com.oceanbase.jdbc.internal.io.input.ReadAheadBufferedStream.fillBuffer(ReadAheadBufferedStream.java:148) at com.oceanbase.jdbc.internal.io.input.ReadAheadBufferedStream.read(ReadAheadBufferedStream.java:121) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.readFully(Ob20PacketInputStream.java:174) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.resolveOb20Packet(Ob20PacketInputStream.java:147) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacketArray(Ob20PacketInputStream.java:97) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacket(Ob20PacketInputStream.java:90) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2211) ... 42 common frames omitted 2025-04-08 20:48:43.252 INFO 3009027 --- [SpringApplicationShutdownHook,,] c.o.ocp.monitor.OcpMonitorManager : monitor manager destroy... 2025-04-08 20:48:43.252 INFO 3009027 --- [SpringApplicationShutdownHook,,] c.o.ocp.monitor.OcpMonitorManager : monitor manager destroyed 2025-04-08 20:48:43.269 INFO 3009027 --- [SpringApplicationShutdownHook,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon stopped 2025-04-08 20:48:43.279 INFO 3009027 --- [SpringApplicationShutdownHook,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Destroy RateLimitFacadeService executor... 2025-04-08 20:48:43.280 INFO 3009027 --- [SpringApplicationShutdownHook,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : RateLimitFacadeService executor destroyed 2025-04-08 20:48:43.282 WARN 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.pool.ProxyConnection : metadb-connect-pool - Connection com.oceanbase.jdbc.OceanBaseConnection@71129d3e marked as broken because of SQLSTATE(08000), ErrorCode(0) java.sql.SQLNonTransientConnectionException: (conn=3221489515) unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:122) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:205) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:967) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:650) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:962) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1017) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1027) at com.oceanbase.ocp.core.distributed.service.DefaultServiceUpdater.close(DefaultServiceUpdater.java:75) at com.oceanbase.ocp.core.distributed.service.DistributedServiceManager.lambda$close$3(DistributedServiceManager.java:102) at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4705) at com.oceanbase.ocp.core.distributed.service.DistributedServiceManager.close(DistributedServiceManager.java:102) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:325) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:259) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1163) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1156) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1120) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1086) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1032) at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145) at java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 32 common frames omitted Caused by: java.sql.SQLNonTransientConnectionException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2730) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2213) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 33 common frames omitted Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.readFully(Ob20PacketInputStream.java:176) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.resolveOb20Packet(Ob20PacketInputStream.java:147) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacketArray(Ob20PacketInputStream.java:97) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacket(Ob20PacketInputStream.java:90) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2211) ... 35 common frames omitted 2025-04-08 20:48:43.283 WARN 3009027 --- [SpringApplicationShutdownHook,,] o.s.b.f.support.DisposableBeanAdapter : Custom destroy method 'close' on bean with name 'distributedService' threw an exception: org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL [DELETE FROM DISTRIBUTED_SERVICE WHERE NAME = ? AND SERVER_ID = ?]; (conn=3221489515) unexpected end of stream, read 0 bytes from 31 (socket was closed by server); nested exception is java.sql.SQLNonTransientConnectionException: (conn=3221489515) unexpected end of stream, read 0 bytes from 31 (socket was closed by server) 2025-04-08 20:48:43.337 INFO 3009027 --- [SpringApplicationShutdownHook,,] c.o.o.alarm.utils.task.AlarmTaskManager : cancelTask task, taskInfo=TaskInfo(id=1, uuid=254bed2a-3c53-4ecf-b850-9c2422739b3a, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=EXECUTING, executorName=192.168.1.61:3009027, createAt=2025-04-08T12:21:34.832298Z, startAt=2025-04-08T12:25:23.094Z, endAt=null, heartbeatAt=2025-04-08T12:25:23.094Z, result=executing) 2025-04-08 20:48:43.339 ERROR 3009027 --- [SpringApplicationShutdownHook,,] c.o.o.alarm.utils.task.AlarmTaskManager : ExecutionException: java.util.concurrent.ExecutionException: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.TransactionException: JDBC begin transaction failed: at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at com.oceanbase.ocp.alarm.utils.task.AlarmTaskManager.removeHasDone(AlarmTaskManager.java:142) at com.oceanbase.ocp.alarm.utils.task.AlarmTaskManager.exit(AlarmTaskManager.java:71) at com.oceanbase.ocp.alarm.schedule.AlarmScheduledJobs.destroy(AlarmScheduledJobs.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeDestroyMethods(InitDestroyAnnotationBeanPostProcessor.java:347) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeDestruction(InitDestroyAnnotationBeanPostProcessor.java:177) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:197) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1163) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1156) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1120) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1086) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1032) at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145) at java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114) at java.lang.Thread.run(Thread.java:750) Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.TransactionException: JDBC begin transaction failed: at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467) at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at com.sun.proxy.$Proxy380.findAll(Unknown Source) at com.oceanbase.ocp.alarm.service.OcpAlarmChannelService.listAll(OcpAlarmChannelService.java:355) at com.oceanbase.ocp.alarm.service.OcpAlarmChannelService$$FastClassBySpringCGLIB$$e4f7fa3b.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.alarm.service.OcpAlarmChannelService$$EnhancerBySpringCGLIB$$98377f9f.listAll() at com.oceanbase.ocp.alarm.core.aggregate.AggregatorFactory.refresh(AggregatorFactory.java:95) at com.oceanbase.ocp.alarm.core.process.AlarmProcessor.process(AlarmProcessor.java:105) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.alarm.core.task.AlarmProcessTask.processOnce(AlarmProcessTask.java:40) at com.oceanbase.ocp.alarm.utils.task.AbstractIntervalTask.call(AbstractIntervalTask.java:41) at com.oceanbase.ocp.alarm.utils.task.AbstractIntervalTask.call(AbstractIntervalTask.java:15) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 common frames omitted Caused by: org.hibernate.TransactionException: JDBC begin transaction failed: at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:78) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:285) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246) at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:164) at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421) ... 32 common frames omitted Caused by: java.sql.SQLException: Connection is closed at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:515) at com.sun.proxy.$Proxy251.setAutoCommit(Unknown Source) at com.zaxxer.hikari.pool.ProxyConnection.setAutoCommit(ProxyConnection.java:414) at com.zaxxer.hikari.pool.HikariProxyConnection.setAutoCommit(HikariProxyConnection.java) at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:72) ... 37 common frames omitted 2025-04-08 20:48:43.340 WARN 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.pool.ProxyConnection : metadb-connect-pool - Connection com.oceanbase.jdbc.OceanBaseConnection@71fd8bf4 marked as broken because of SQLSTATE(08000), ErrorCode(0) java.sql.SQLNonTransientConnectionException: (conn=3221489517) unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:122) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeQuery(JDBC4PreparedStatement.java:185) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57) at org.hibernate.loader.Loader.getResultSet(Loader.java:2322) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2075) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2037) at org.hibernate.loader.Loader.doQuery(Loader.java:956) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357) at org.hibernate.loader.Loader.doList(Loader.java:2868) at org.hibernate.loader.Loader.doList(Loader.java:2850) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682) at org.hibernate.loader.Loader.list(Loader.java:2677) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:540) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:218) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1459) at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617) at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1665) at sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.orm.jpa.SharedEntityManagerCreator$DeferredQueryInvocationHandler.invoke(SharedEntityManagerCreator.java:410) at com.sun.proxy.$Proxy328.getSingleResult(Unknown Source) at org.springframework.data.jpa.repository.query.JpaQueryExecution$SingleEntityExecution.doExecute(JpaQueryExecution.java:198) at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:90) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:156) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:144) at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137) at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121) at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:160) at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:139) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:81) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at com.sun.proxy.$Proxy390.fetchTasksByUUID(Unknown Source) at com.oceanbase.ocp.alarm.service.TaskRepositoryService.endTask(TaskRepositoryService.java:122) at com.oceanbase.ocp.alarm.service.TaskRepositoryService$$FastClassBySpringCGLIB$$78ac8e72.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.alarm.service.TaskRepositoryService$$EnhancerBySpringCGLIB$$653c16a4.endTask() at com.oceanbase.ocp.alarm.utils.task.AlarmTaskManager.removeHasDone(AlarmTaskManager.java:152) at com.oceanbase.ocp.alarm.utils.task.AlarmTaskManager.exit(AlarmTaskManager.java:71) at com.oceanbase.ocp.alarm.schedule.AlarmScheduledJobs.destroy(AlarmScheduledJobs.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeDestroyMethods(InitDestroyAnnotationBeanPostProcessor.java:347) at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeDestruction(InitDestroyAnnotationBeanPostProcessor.java:177) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:197) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1163) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1156) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1120) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1086) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1032) at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145) at java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 80 common frames omitted Caused by: java.sql.SQLNonTransientConnectionException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2730) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2213) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 81 common frames omitted Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.readFully(Ob20PacketInputStream.java:176) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.resolveOb20Packet(Ob20PacketInputStream.java:147) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacketArray(Ob20PacketInputStream.java:97) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacket(Ob20PacketInputStream.java:90) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2211) ... 83 common frames omitted 2025-04-08 20:48:43.341 WARN 3009027 --- [SpringApplicationShutdownHook,,] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08000 2025-04-08 20:48:43.341 ERROR 3009027 --- [SpringApplicationShutdownHook,,] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=3221489517) unexpected end of stream, read 0 bytes from 31 (socket was closed by server) 2025-04-08 20:48:43.347 WARN 3009027 --- [SpringApplicationShutdownHook,,] .s.c.a.CommonAnnotationBeanPostProcessor : Destroy method on bean with name 'alarmScheduledJobs' threw an exception: org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet 2025-04-08 20:48:43.360 INFO 3009027 --- [SpringApplicationShutdownHook,,] c.o.ocp.core.sharding.ShardingFactory : Shutdown sharding factory, coordinatorSize=3, consumerSize=3 2025-04-08 20:48:43.362 WARN 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.pool.ProxyConnection : metadb-connect-pool - Connection com.oceanbase.jdbc.OceanBaseConnection@16902d2d marked as broken because of SQLSTATE(08000), ErrorCode(0) com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:329) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:298) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.setAutoCommit(AbstractQueryProtocol.java:1939) at com.oceanbase.jdbc.OceanBaseConnection.setAutoCommit(OceanBaseConnection.java:1080) at com.zaxxer.hikari.pool.ProxyConnection.setAutoCommit(ProxyConnection.java:414) at com.zaxxer.hikari.pool.HikariProxyConnection.setAutoCommit(HikariProxyConnection.java) at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:72) at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:285) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246) at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:164) at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421) at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.distributed.lock.jdbc.JdbcLockRepository.close(JdbcLockRepository.java:74) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:242) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1163) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1156) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1120) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1086) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1032) at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145) at java.lang.Iterable.forEach(Iterable.java:75) at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLNonTransientConnectionException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2730) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2213) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:324) ... 30 common frames omitted Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 31 (socket was closed by server) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.readFully(Ob20PacketInputStream.java:176) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.resolveOb20Packet(Ob20PacketInputStream.java:147) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacketArray(Ob20PacketInputStream.java:97) at com.oceanbase.jdbc.internal.io.input.Ob20PacketInputStream.getPacket(Ob20PacketInputStream.java:90) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2211) ... 32 common frames omitted 2025-04-08 20:48:43.362 WARN 3009027 --- [SpringApplicationShutdownHook,,] o.s.b.f.support.DisposableBeanAdapter : Invocation of close method failed on bean with name 'lockRepository': org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.TransactionException: JDBC begin transaction failed: 2025-04-08 20:48:43.366 INFO 3009027 --- [SpringApplicationShutdownHook,,] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:48:43.370 INFO 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Shutdown initiated... 2025-04-08 20:48:43.376 INFO 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Shutdown completed. 2025-04-08 20:48:43.378 INFO 3009027 --- [SpringApplicationShutdownHook,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} closing ... 2025-04-08 20:48:43.383 INFO 3009027 --- [SpringApplicationShutdownHook,,] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} closed 2025-04-08 20:48:43.389 INFO 3009027 --- [SpringApplicationShutdownHook,,] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:48:43.393 WARN 3009027 --- [ocp-updater-0,,] com.zaxxer.hikari.pool.PoolBase : metadb-connect-pool - Failed to validate connection com.oceanbase.jdbc.OceanBaseConnection@248c900e ((conn=3221489522) Connection.setNetworkTimeout cannot be called on a closed connection). Possibly consider using a shorter maxLifetime value. 2025-04-08 20:48:43.394 INFO 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Shutdown initiated... 2025-04-08 20:48:43.394 WARN 3009027 --- [ocp-updater-0,,] com.zaxxer.hikari.pool.PoolBase : metadb-connect-pool - Failed to validate connection com.oceanbase.jdbc.OceanBaseConnection@14e1f9c7 ((conn=3221489519) Connection.setNetworkTimeout cannot be called on a closed connection). Possibly consider using a shorter maxLifetime value. 2025-04-08 20:48:43.396 INFO 3009027 --- [SpringApplicationShutdownHook,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Shutdown completed. 2025-04-08 20:58:12.615 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:58:12.615 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:58:12.615 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:58:12.615 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:58:12.615 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:58:12.616 INFO 16270 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2025-04-08 20:58:12.624 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:58:18.293 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:58:18.294 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:58:19.765 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1447 ms. Found 173 JPA repository interfaces. 2025-04-08 20:58:19.772 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:58:19.772 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:58:19.967 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 194 ms. Found 16 JPA repository interfaces. 2025-04-08 20:58:19.967 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 20:58:19.967 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 20:58:20.013 INFO 16270 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 46 ms. Found 5 JPA repository interfaces. 2025-04-08 20:58:22.886 INFO 16270 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=535a2406-c3b5-3b8b-8149-b06173f75dfb 2025-04-08 20:58:23.704 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2bdb5e0f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:23.967 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2bdb5e0f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.004 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#45f9d394' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.018 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#d28c214' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.028 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#d28c214' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.056 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration' of type [org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.072 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metadbConfiguration' of type [com.oceanbase.ocp.server.common.config.MetadbConfiguration$$EnhancerBySpringCGLIB$$d34656c4] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.103 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration' of type [org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.113 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.120 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'loadBalancerClientsDefaultsMappingsProvider' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration$$Lambda$661/24226406] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.159 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultsBindHandlerAdvisor' of type [org.springframework.cloud.commons.config.DefaultsBindHandlerAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.180 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpClusterProperty.MetaDbProperties' of type [com.oceanbase.ocp.core.config.OcpClusterProperty$MetaDbProperties$$EnhancerBySpringCGLIB$$6b8aa4d3] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.196 INFO 16270 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:58:24.196 INFO 16270 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2025-04-08 20:58:24.196 INFO 16270 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:58:24.202 INFO 16270 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2025-04-08 20:58:24.253 INFO 16270 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2025-04-08 20:58:24.262 INFO 16270 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:58:24.262 INFO 16270 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:58:24.262 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaDataSource' of type [com.zaxxer.hikari.HikariDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.308 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.sql.init-org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties' of type [org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.334 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceScriptDatabaseInitializer' of type [org.springframework.boot.autoconfigure.sql.init.SqlDataSourceScriptDatabaseInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.363 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.376 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa.hibernate-org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.393 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.407 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hikariPoolDataSourceMetadataProvider' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration$$Lambda$429/1370224403] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.472 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.530 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaVendorAdapter' of type [org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.555 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'entityManagerFactoryBuilder' of type [org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:24.752 INFO 16270 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 20:58:25.071 INFO 16270 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2025-04-08 20:58:25.852 INFO 16270 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2025-04-08 20:58:29.299 INFO 16270 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:58:30.540 WARN 16270 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2025-04-08 20:58:31.426 WARN 16270 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 20:58:31.426 WARN 16270 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 20:58:34.800 INFO 16270 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 20:58:34.867 INFO 16270 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:58:34.884 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:34.901 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [com.sun.proxy.$Proxy294] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.025 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1b475663' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.234 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'propertyManager.MonitorDbExtraProperties' of type [com.oceanbase.ocp.core.property.PropertyManager$MonitorDbExtraProperties$$EnhancerBySpringCGLIB$$28412866] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.240 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'monitordbConfiguration' of type [com.oceanbase.ocp.server.common.config.MonitordbConfiguration$$EnhancerBySpringCGLIB$$11155b5b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.310 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpAzoneModelConverter' of type [com.oceanbase.ocp.ha.maz.OcpAzoneModelConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.329 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2376463f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.330 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2376463f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.331 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6a8b30a3' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.331 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#39fb9d55' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.332 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#39fb9d55' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.333 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#62eb0507' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.403 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.repository.config.JpaMetamodelMappingContextFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:35.412 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.mapping.JpaMetamodelMappingContext] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.344 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.355 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [com.sun.proxy.$Proxy300] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.415 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#48e1928a' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.416 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#48e1928a' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.417 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#579ffb7e' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.418 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#333674ad' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.419 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#333674ad' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.420 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5a79b792' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.778 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.778 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [com.sun.proxy.$Proxy304] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.802 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#119a8125' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.802 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#119a8125' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.803 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#156eb310' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.803 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#614876bc' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.804 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#614876bc' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.805 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#798ba61' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.831 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.832 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [com.sun.proxy.$Proxy305] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.856 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#781223fc' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.857 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#781223fc' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.857 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#d78c346' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.858 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#51fb4092' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.858 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#51fb4092' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.860 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7a3e0c5' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.909 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.910 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [com.sun.proxy.$Proxy306] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.967 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e3b72ad' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.967 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e3b72ad' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.968 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7e7cb19c' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.969 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7767c18d' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.969 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7767c18d' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.972 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3dacb1e' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.997 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:36.998 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [com.sun.proxy.$Proxy307] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.085 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaTransactionManager' of type [org.springframework.orm.jpa.JpaTransactionManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.385 INFO 16270 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action group type info: {"HOST_MANAGE":"HOST","ALL":"UNKNOWN","HOST_MAINTAIN":"HOST","ALL_READ":"UNKNOWN","OBPROXY_MAINTAIN":"OBPROXY","ORGANIZATION_READ":"ORGANIZATION","HOST_CREATE":"HOST","USER_MANAGE":"USER","ARBITRATION_READ":"ARBITRATION","ARBITRATION_CREATE":"ARBITRATION","BINLOG_MANAGE":"BINLOG","OBPROXY_MANAGE":"OBPROXY","HOST_READ":"HOST","BINLOG_READ":"BINLOG","AUDIT_READ":"AUDIT","USER_READ":"USER","ROLE_READ":"ROLE","CLUSTER_MANAGE":"CLUSTER","CLUSTER_CREATE":"CLUSTER","TENANT_MANAGE":"TENANT","CLUSTER_MAINTAIN":"CLUSTER","TENANT_MAINTAIN":"TENANT","CLUSTER_READ":"CLUSTER","BINLOG_MAINTAIN":"BINLOG","TENANT_READ":"TENANT","OBPROXY_READ":"OBPROXY","ARBITRATION_MAINTAIN":"ARBITRATION","ROLE_MANAGE":"ROLE","BINLOG_CREATE":"BINLOG","ARBITRATION_MANAGE":"ARBITRATION","OBPROXY_CREATE":"OBPROXY"} 2025-04-08 20:58:37.386 INFO 16270 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action type info: {"ALARM:CREATE":"ALARM","SELFCURE:UPDATE":"SELFCURE","HOST:DELETE":"HOST","ROLE:CREATE":"ROLE","HOST:UPDATE":"HOST","CLUSTER:UPDATE":"CLUSTER","INSPECTION:UPDATE":"INSPECTION","CLUSTER:READ":"CLUSTER","PACKAGE:READ":"PACKAGE","ALARM:READ":"ALARM","TAG:DELETE":"TAG","BINLOG:CREATE":"BINLOG","ARBITRATION:CREATE":"ARBITRATION","SELFCURE:READ":"SELFCURE","*":"UNKNOWN","CLUSTER:DELETE":"CLUSTER","CREDENTIAL:DELETE":"CREDENTIAL","OBPROXY:READ":"OBPROXY","PROPERTY:READ":"PROPERTY","ARBITRATION:UPDATE":"ARBITRATION","OBPROXY:DELETE":"OBPROXY","TENANT:UPDATE":"TENANT","BINLOG:READ":"BINLOG","HOST:READ":"HOST","INSPECTION:CREATE":"INSPECTION","ALARM:UPDATE":"ALARM","PROPERTY:UPDATE":"PROPERTY","BINLOG:UPDATE":"BINLOG","PACKAGE:UPDATE":"PACKAGE","HOST:CREATE":"HOST","ROLE:DELETE":"ROLE","TAG:READ":"TAG","ROLE:READ":"ROLE","BINLOG:DELETE":"BINLOG","TASK:UPDATE":"TASK","USER:DELETE":"USER","CREDENTIAL:UPDATE":"CREDENTIAL","USER:UPDATE":"USER","TENANT:READ":"TENANT","TAG:UPDATE":"TAG","OBPROXY:CREATE":"OBPROXY","ARBITRATION:READ":"ARBITRATION","ROLE:UPDATE":"ROLE","ORGANIZATION:READ":"ORGANIZATION","TENANT:DELETE":"TENANT","USER:READ":"USER","AUDIT:READ":"AUDIT","TASK:READ":"TASK","TAG:CREATE":"TAG","INSPECTION:READ":"INSPECTION","ARBITRATION:DELETE":"ARBITRATION","USER:CREATE":"USER","PACKAGE:DELETE":"PACKAGE","ALARM:DELETE":"ALARM","CREDENTIAL:READ":"CREDENTIAL","OBPROXY:UPDATE":"OBPROXY","CREDENTIAL:CREATE":"CREDENTIAL","CLUSTER:CREATE":"CLUSTER"} 2025-04-08 20:58:37.386 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionMetaHelper' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ActionMetaHelper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.412 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultPerfConfiguration' of type [com.oceanbase.ocp.server.common.config.DefaultPerfConfiguration$$EnhancerBySpringCGLIB$$b069a969] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.446 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'authenticationFacadeImpl' of type [com.oceanbase.ocp.core.security.AuthenticationFacadeImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.468 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4717bdb3' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.469 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4717bdb3' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.469 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6da69c15' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.470 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4fda9835' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.470 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4fda9835' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.471 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#554159c5' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.550 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.551 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [com.sun.proxy.$Proxy310] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.576 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOperator' of type [com.oceanbase.ocp.core.security.organization.operator.HostOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.598 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f32afbe' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.599 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f32afbe' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.600 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#196a99d0' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.601 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c0f82f' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.601 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c0f82f' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.604 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#16d76df7' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.672 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.673 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [com.sun.proxy.$Proxy312] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.688 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObClusterOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.704 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#356eb8d4' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.704 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#356eb8d4' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.705 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#71e28a63' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.705 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3fdc7786' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.706 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3fdc7786' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.707 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6af2203' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.766 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.767 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [com.sun.proxy.$Proxy313] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.786 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObproxyOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.803 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#399f5a98' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.803 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#399f5a98' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.804 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#342daa77' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.804 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3fa4fe1f' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.805 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3fa4fe1f' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.806 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2c2a8c3d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.842 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.842 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [com.sun.proxy.$Proxy315] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.856 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObTenantOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.873 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b349ea9' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.873 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b349ea9' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.874 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d74b85d' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.875 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ee7e602' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.875 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ee7e602' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.876 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3c4b9a8f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.941 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.942 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [com.sun.proxy.$Proxy316] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.964 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOperator' of type [com.oceanbase.ocp.core.security.organization.operator.UserOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.979 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d8f4623' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.979 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d8f4623' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.980 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27320a39' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.980 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#597ca350' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.981 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#597ca350' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:37.982 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3425709a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.004 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.005 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [com.sun.proxy.$Proxy318] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.023 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ArbitrationOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.039 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#40fd8aa1' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.040 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#40fd8aa1' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.040 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43fe629a' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.041 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2aeef1ca' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.041 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2aeef1ca' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.042 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4517959f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.078 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.079 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [com.sun.proxy.$Proxy319] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.093 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogOperator' of type [com.oceanbase.ocp.core.security.organization.operator.BinlogOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.094 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceManager' of type [com.oceanbase.ocp.core.security.organization.ResourceManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.109 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#292e8d0a' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.110 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#292e8d0a' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.111 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#41566d56' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.111 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7bd9e2f7' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.111 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7bd9e2f7' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.113 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#703aca0d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.151 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.152 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [com.sun.proxy.$Proxy320] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.181 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7458d7a9' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.181 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7458d7a9' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.182 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3aabf6ae' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.183 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#615a5152' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.183 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#615a5152' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.184 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5e9feddd' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.204 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.204 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [com.sun.proxy.$Proxy322] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.238 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7c671bcb' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.239 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7c671bcb' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.240 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2fe4a108' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.241 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75806ed6' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.241 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75806ed6' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.244 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6bd9d056' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.271 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.271 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [com.sun.proxy.$Proxy323] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.291 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'getOrganizationService' of type [com.oceanbase.ocp.core.security.organization.OrganizationServiceImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.316 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1a312c51' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.316 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1a312c51' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.317 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#407fda21' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.317 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#67ab557b' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.318 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#67ab557b' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.319 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6fb38ea5' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.419 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.419 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [com.sun.proxy.$Proxy324] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.453 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#493658fa' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.453 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#493658fa' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.454 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#66c10c43' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.455 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7b7d5d8d' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.455 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7b7d5d8d' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.456 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5aea3e4e' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.485 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.485 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [com.sun.proxy.$Proxy325] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.511 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#48d54b1c' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.511 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#48d54b1c' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.512 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5d7225b5' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.512 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6291cdb4' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.513 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6291cdb4' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.514 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#797168d9' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.572 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.573 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [com.sun.proxy.$Proxy326] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.606 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#d55b14b' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.606 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#d55b14b' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.607 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5f87417a' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.607 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4b3816c' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.607 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4b3816c' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.608 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#47a96f6a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.638 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.638 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [com.sun.proxy.$Proxy327] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.666 INFO 16270 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 20:58:38.945 INFO 16270 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 20:58:38.945 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.972 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#64acc538' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.973 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#64acc538' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.974 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#447d14a9' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.974 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4f880f30' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.975 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4f880f30' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:38.976 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74aa2842' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.018 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.019 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [com.sun.proxy.$Proxy330] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.048 INFO 16270 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 20:58:39.066 INFO 16270 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 20:58:39.066 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obProxyQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObProxyQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.083 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#467c9ad9' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.083 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#467c9ad9' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.084 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#48d74bb4' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.084 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3fad2065' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.084 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3fad2065' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.085 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#79703b86' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.106 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.107 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [com.sun.proxy.$Proxy331] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.134 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#40bf36b1' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.134 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#40bf36b1' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.135 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#25bfc73' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.135 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4642b0f8' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.136 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4642b0f8' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.137 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#9d45e6e' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.175 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.176 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [com.sun.proxy.$Proxy332] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.199 INFO 16270 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 20:58:39.223 INFO 16270 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 20:58:39.223 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogQueryCache' of type [com.oceanbase.ocp.core.cache.helper.BinlogQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.226 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceCache' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ResourceCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.228 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.239 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'adminResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.249 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'clusterResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.259 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.269 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.278 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.287 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'tenantResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.289 INFO 16270 --- [main,,] c.o.o.c.s.a.r.ResourceSupplierManager : Resource supplier list: [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl@cf051f1, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl@3a4f4a81, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl@7c26f301, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl@21c1fb2e, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl@39a95375, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl@7082d3e2, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl@291be813] 2025-04-08 20:58:39.289 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceSupplierManager' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.ResourceSupplierManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.316 INFO 16270 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$df5625f1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 20:58:39.330 INFO 16270 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2025-04-08 20:58:39.801 INFO 16270 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=192.168.1.61:8080, name=b88d8e1c7d 2025-04-08 20:58:39.834 WARN 16270 --- [main,,] c.o.o.c.d.server.DefaultServerUpdater : Deleted expired server: DistributedServer{id=192.168.1.61:8080, name=b88d8e1c7d, expireInSeconds=30, createTime=2025-04-08T20:23:16+08:00, updateTime=2025-04-08T20:48:38+08:00, expireTime=2025-04-08T20:49:08+08:00, isActive=false} 2025-04-08 20:58:39.891 INFO 16270 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 192.168.1.61:8080 2025-04-08 20:58:40.711 INFO 16270 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2025-04-08 20:58:40.741 INFO 16270 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2025-04-08 20:58:40.741 INFO 16270 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.98] 2025-04-08 20:58:40.942 INFO 16270 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2025-04-08 20:58:40.942 INFO 16270 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 28318 ms 2025-04-08 20:58:41.304 INFO 16270 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2025-04-08 20:58:41.553 INFO 16270 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:58:41.553 INFO 16270 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2025-04-08 20:58:41.553 INFO 16270 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:58:41.583 INFO 16270 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:58:41.583 INFO 16270 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:58:42.056 INFO 16270 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2025-04-08 20:58:42.066 INFO 16270 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : host 192.168.1.61, port 2881, database monitor_database 2025-04-08 20:58:42.067 INFO 16270 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 20:58:42.067 INFO 16270 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2025-04-08 20:58:42.067 INFO 16270 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2025-04-08 20:58:42.067 INFO 16270 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2025-04-08 20:58:42.081 INFO 16270 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2025-04-08 20:58:42.082 INFO 16270 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 20:58:42.082 INFO 16270 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 20:58:42.704 INFO 16270 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 20:58:44.944 INFO 16270 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:58:45.334 INFO 16270 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 20:58:45.335 INFO 16270 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 20:58:58.773 INFO 16270 --- [main,,] org.reflections.Reflections : Reflections took 4736 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 20:59:03.078 INFO 16270 --- [main,,] org.reflections.Reflections : Reflections took 4242 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 20:59:07.703 INFO 16270 --- [main,,] org.reflections.Reflections : Reflections took 4615 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 20:59:07.720 INFO 16270 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2025-04-08 20:59:07.892 INFO 16270 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2025-04-08 20:59:08.359 INFO 16270 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2025-04-08 20:59:08.946 INFO 16270 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2025-04-08 20:59:08.974 INFO 16270 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2025-04-08 20:59:08.974 INFO 16270 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2025-04-08 20:59:09.818 INFO 16270 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=arbitration_group_auto_replace_failed,ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,arbitration_auto_replace_operation_info,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_merge_error,ob_tenant_name_mismatch, 2025-04-08 20:59:10.767 INFO 16270 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table switch time is 1744114820 2025-04-08 20:59:11.958 INFO 16270 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 20:59:12.024 INFO 16270 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 20:59:12.034 INFO 16270 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2025-04-08 20:59:13.003 INFO 16270 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:59:12.667+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion 2025-04-08 20:59:13.009 INFO 16270 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:59:13.009 INFO 16270 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T20:59:12.667+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion 2025-04-08 20:59:13.009 INFO 16270 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 20:59:13.010 INFO 16270 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 20:59:13.476 INFO 16270 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/root/ocp-server/alarm 2025-04-08 20:59:13.641 INFO 16270 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2025-04-08 20:59:13.689 INFO 16270 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=192.168.1.61:16270 2025-04-08 20:59:13.770 INFO 16270 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 20:59:13.880 INFO 16270 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 20:59:13.899 INFO 16270 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2025-04-08 20:59:13.941 INFO 16270 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2025-04-08 20:59:14.197 INFO 16270 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2025-04-08 20:59:14.200 INFO 16270 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2025-04-08 20:59:14.270 INFO 16270 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=2, heartbeatTime=2025-04-08T20:48:41+08:00, now=2025-04-08T20:59:14.269+08:00 2025-04-08 20:59:14.273 INFO 16270 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=1, heartbeatTime=2025-04-08T20:48:41+08:00, now=2025-04-08T20:59:14.273+08:00 2025-04-08 20:59:14.481 INFO 16270 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=192.168.1.61:8080, consumerId=2, resources=[ShardingSlice(consumerId=2, type=DEFAULT, resourceIds=[18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 80, 81, 106, 107, 250, 251, 252, 253, 254])] 2025-04-08 20:59:14.481 INFO 16270 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-target, name=192.168.1.61:8080, consumerId=1, resources=[ShardingSlice(consumerId=1, type=OBCluster, resourceIds=[2]), ShardingSlice(consumerId=1, type=Host, resourceIds=[2])] 2025-04-08 20:59:14.571 INFO 16270 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2025-04-08 20:59:15.082 INFO 16270 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2025-04-08 20:59:15.142 INFO 16270 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2025-04-08 20:59:15.332 INFO 16270 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/root/software 2025-04-08 20:59:17.484 INFO 16270 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from OFFLINE to NEW. 2025-04-08 20:59:17.714 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2025-04-08 20:59:17.715 INFO 16270 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2025-04-08 20:59:17.716 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2025-04-08 20:59:17.733 INFO 16270 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2025-04-08 20:59:17.734 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2025-04-08 20:59:17.734 INFO 16270 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2025-04-08 20:59:17.736 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2025-04-08 20:59:17.736 INFO 16270 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2025-04-08 20:59:17.737 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2025-04-08 20:59:17.737 INFO 16270 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2025-04-08 20:59:17.738 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2025-04-08 20:59:17.738 INFO 16270 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2025-04-08 20:59:17.740 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2025-04-08 20:59:17.740 INFO 16270 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2025-04-08 20:59:17.741 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2025-04-08 20:59:17.744 INFO 16270 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2025-04-08 20:59:17.745 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2025-04-08 20:59:17.745 INFO 16270 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2025-04-08 20:59:17.746 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2025-04-08 20:59:17.746 INFO 16270 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2025-04-08 20:59:17.747 INFO 16270 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2025-04-08 20:59:17.747 INFO 16270 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2025-04-08 20:59:19.470 INFO 16270 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=2, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@6c3f4afc 2025-04-08 20:59:19.472 INFO 16270 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:59:19.478 INFO 16270 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=1, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@3dbe57d5 2025-04-08 20:59:19.478 INFO 16270 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:59:20.509 INFO 16270 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:59:20.516 INFO 16270 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:59:21.082 INFO 16270 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:59:21.094 INFO 16270 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:59:23.521 INFO 16270 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:59:23.521 INFO 16270 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 20:59:23.527 INFO 16270 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:59:23.527 INFO 16270 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 20:59:23.952 INFO 16270 --- [main,,] org.reflections.Reflections : Reflections took 4202 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 20:59:23.963 INFO 16270 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2025-04-08 20:59:23.964 INFO 16270 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2025-04-08 20:59:23.964 INFO 16270 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2025-04-08 20:59:23.964 INFO 16270 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2025-04-08 20:59:23.964 INFO 16270 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2025-04-08 20:59:23.981 INFO 16270 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on b88d8e1c7d 2025-04-08 20:59:24.899 INFO 16270 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2025-04-08 20:59:25.536 INFO 16270 --- [main,,] c.o.o.s.g.prefs.PrefsParameterService : Start loading prefs template. 2025-04-08 20:59:25.958 INFO 16270 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Begin init binlog cluster parameter info. 2025-04-08 20:59:25.973 INFO 16270 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Finish init binlog cluster parameter info, parameter blacklist: [] 2025-04-08 20:59:26.021 INFO 16270 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Begin init binlog instance parameter info. 2025-04-08 20:59:26.031 INFO 16270 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Finish init binlog instance parameter info, parameter blacklist: [] 2025-04-08 20:59:26.940 INFO 16270 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2025-04-08 20:59:27.941 INFO 16270 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2025-04-08 20:59:27.972 INFO 16270 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB, valid=true 2025-04-08 20:59:27.972 INFO 16270 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Sensitive data encryption key is legal. 2025-04-08 20:59:28.077 INFO 16270 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2025-04-08 20:59:28.339 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 20:59:28.376 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 20:59:28.396 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:59:28.404 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:59:29.660 INFO 16270 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 192.168.1.61:8080 2025-04-08 20:59:29.669 INFO 16270 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2025-04-08 20:59:29.716 WARN 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 20:59:29.716 INFO 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 20:59:30.078 INFO 16270 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2025-04-08 20:59:30.092 INFO 16270 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2025-04-08 20:59:31.610 INFO 16270 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/root/ocp/ocp-server/lib 2025-04-08 20:59:31.610 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2025-04-08 20:59:31.652 INFO 16270 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=obproxyd.sh, objectId=7 2025-04-08 20:59:31.676 INFO 16270 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2025-04-08 20:59:31.707 INFO 16270 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh.sha1sum 2025-04-08 20:59:31.707 INFO 16270 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh 2025-04-08 20:59:31.707 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2025-04-08 20:59:31.758 INFO 16270 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=1000001 2025-04-08 20:59:31.851 INFO 16270 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=1000001, name=obproxyd.sh 2025-04-08 20:59:31.864 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 20:59:31.898 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 20:59:31.898 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 20:59:31.901 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 20:59:31.901 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:59:31.904 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:59:31.904 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:59:31.907 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 20:59:31.907 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 20:59:31.909 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 20:59:31.911 INFO 16270 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.5-20250319105844 2025-04-08 20:59:31.919 INFO 16270 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.5-20250319105844, updatedValue=4.3.5-20250319105844, runningValue=4.3.5-20250319105844 2025-04-08 20:59:31.960 INFO 16270 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"update_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition","timeoutNs":0},"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event","timeoutNs":0},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception","timeoutNs":0},"cleanStorageObjectBlock":{"batchIntervalMillis":10000,"batchSize":10,"condition":"file_id not in (select id from storage_object_meta)","priority":0,"tableName":"storage_object_block","timeoutNs":0}} 2025-04-08 20:59:32.383 INFO 16270 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2025-04-08 20:59:32.623 INFO 16270 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2025-04-08 20:59:32.626 INFO 16270 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2025-04-08 20:59:32.716 WARN 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 20:59:32.716 INFO 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 20:59:33.831 INFO 16270 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2025-04-08 20:59:33.930 INFO 16270 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2025-04-08 20:59:33.931 INFO 16270 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117200, interval=60 2025-04-08 20:59:33.933 INFO 16270 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2025-04-08 20:59:33.968 INFO 16270 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2025-04-08 20:59:34.193 INFO 16270 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2025-04-08 20:59:34.444 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 20:59:34.450 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 20:59:34.452 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:59:34.453 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 20:59:34.453 INFO 16270 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2025-04-08 20:59:34.525 INFO 16270 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2025-04-08 20:59:34.532 INFO 16270 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2025-04-08 20:59:34.585 INFO 16270 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2025-04-08 20:59:34.602 INFO 16270 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2025-04-08 20:59:34.667 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=CMDB_SYNCER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:48:38+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.669 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=MC_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:48:38+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.670 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PERF_SQL_ROLLUP, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:48:31+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.672 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:41:15+08:00, updateTime=2025-04-08T20:48:15+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.674 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=RATE_LIMIT, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:48:39+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.675 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:48:15+08:00, updateTime=2025-04-08T20:48:15+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.684 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:40:45+08:00, updateTime=2025-04-08T20:48:15+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.685 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=TASK_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:24:13+08:00, updateTime=2025-04-08T20:48:38+08:00, expireTime=null, isLeader=false} 2025-04-08 20:59:34.687 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.690 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.692 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.694 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.697 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.698 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.725 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.727 WARN 16270 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 20:59:34.961 INFO 16270 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2025-04-08 20:59:35.109 INFO 16270 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=2, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@7ef7e65c 2025-04-08 20:59:35.109 INFO 16270 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=192.168.1.61:8080, consumerId=3, resources=[ShardingSlice(consumerId=3, type=OB_CLUSTER, resourceIds=[2]), ShardingSlice(consumerId=3, type=HOST_MONITOR, resourceIds=[2])] 2025-04-08 20:59:35.110 INFO 16270 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 20:59:35.606 INFO 16270 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2025-04-08 20:59:35.620 INFO 16270 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2025-04-08 20:59:35.621 INFO 16270 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2025-04-08 20:59:35.625 INFO 16270 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2025-04-08 20:59:35.717 INFO 16270 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 gets the leadership... 2025-04-08 20:59:35.962 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$1985/80735368@465490ee with [org.springframework.security.web.session.DisableEncodeUrlFilter@556ba2a6, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@73a0a1db, org.springframework.security.web.context.SecurityContextPersistenceFilter@4c591b5f, org.springframework.security.web.header.HeaderWriterFilter@64775ea6, org.springframework.security.web.authentication.logout.LogoutFilter@66065ad9, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@3883b27a, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@5a328664, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@52c46116, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@249cacec, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@3366beb8, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@6c602ae4, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@4284cb43, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@1a9da67, org.springframework.security.web.session.SessionManagementFilter@528a6639, org.springframework.security.web.access.ExceptionTranslationFilter@2da901c1, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@378cccc8] 2025-04-08 20:59:35.973 INFO 16270 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2025-04-08 20:59:35.973 INFO 16270 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2025-04-08 20:59:36.167 INFO 16270 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2025-04-08 20:59:36.310 INFO 16270 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2025-04-08 20:59:36.317 INFO 16270 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2025-04-08 20:59:36.330 INFO 16270 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2025-04-08 20:59:37.433 INFO 16270 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2025-04-08 20:59:37.435 INFO 16270 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2025-04-08 20:59:37.725 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@2f7359fc, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@3f15387c, org.springframework.security.web.context.SecurityContextPersistenceFilter@5b9d3744, org.springframework.security.web.header.HeaderWriterFilter@3ab367bd, org.springframework.security.web.csrf.CsrfFilter@663e8537, org.springframework.security.web.authentication.logout.LogoutFilter@6338340a, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@11ebd65a, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@1a45146, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@2c23f7b1, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@69dc41a6, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@478663f, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@797340c8, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@4c7a5aa3, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@70fce8c5, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@6a79b8c8, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@4b56b4b5, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@5a3a5d88, org.springframework.security.web.session.SessionManagementFilter@28b64ae1, org.springframework.security.web.access.ExceptionTranslationFilter@50dc313e, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@52681ac9] 2025-04-08 20:59:37.760 INFO 16270 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2025-04-08 20:59:38.111 INFO 16270 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 20:59:38.132 INFO 16270 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=192.168.1.61:8080, consumerId=3, resources=[] 2025-04-08 20:59:38.162 INFO 16270 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2025-04-08 20:59:38.163 INFO 16270 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 20:59:38.470 WARN 16270 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2025-04-08 20:59:38.717 INFO 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:59:38.728 INFO 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:59:39.346 INFO 16270 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2025-04-08 20:59:41.134 INFO 16270 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 20:59:41.134 INFO 16270 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2025-04-08 20:59:41.139 INFO 16270 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=192.168.1.61:8080, consumerId=3, resources=[ShardingSlice(consumerId=3, type=OB_CLUSTER, resourceIds=[2]), ShardingSlice(consumerId=3, type=HOST_MONITOR, resourceIds=[2])] 2025-04-08 20:59:41.728 INFO 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:59:41.729 INFO 16270 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:59:42.539 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.539 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/static/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/static/**'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff2']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff2'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.ttf']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.ttf'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.tif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.tif'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.gif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.gif'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.jpg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.jpg'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.png']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.png'] 2025-04-08 20:59:42.540 WARN 16270 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 20:59:42.540 INFO 16270 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2025-04-08 20:59:43.279 INFO 16270 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2025-04-08 20:59:43.569 INFO 16270 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=15 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=inhibitAlarms, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.803 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=6 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkSuspendTenantCompactionStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=11 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterCgroupValue, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationServiceGroup, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationAutoReplaceStatusTask, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.804 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.822 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2025-04-08 20:59:43.823 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2025-04-08 20:59:43.823 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.829 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.829 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.829 INFO 16270 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 20:59:43.847 INFO 16270 --- [ocp-schedules-20,61431037cf7db0bf,c8f85ba55672a648] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 20:59:43.848 INFO 16270 --- [ocp-schedules-7,d2042ed4d4f8e909,c7b75962e9a85c15] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 20:59:43.850 INFO 16270 --- [ocp-schedules-25,b19254e5a5518516,fee04aa092419d81] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 20:59:43.850 INFO 16270 --- [ocp-schedules-26,57b0ca43518ac121,11554195b1f6448a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 20:59:43.863 INFO 16270 --- [ocp-schedules-5,8b0708a4c765853f,b04892889d949436] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 20:59:43.867 INFO 16270 --- [ocp-schedules-13,152372a4c5966590,b63b7a5c3e02c09c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 20:59:43.870 INFO 16270 --- [ocp-schedules-21,25e7ef84dee04f56,1dba57d24d59f311] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 20:59:43.871 WARN 16270 --- [ocp-schedules-26,57b0ca43518ac121,11554195b1f6448a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncClusterInfo, confName=syncClusterInfo 2025-04-08 20:59:43.873 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2025-04-08 20:59:43.873 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2025-04-08 20:59:43.873 INFO 16270 --- [ocp-schedules-8,078b52336a7fa4d3,06c92ff2d96951f5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 20:59:43.874 ERROR 16270 --- [ocp-schedules-8,078b52336a7fa4d3,06c92ff2d96951f5] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2025-04-08 20:59:43.875 ERROR 16270 --- [ocp-schedules-26,57b0ca43518ac121,11554195b1f6448a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:80) ... 15 common frames omitted 2025-04-08 20:59:43.876 INFO 16270 --- [ocp-schedules-14,56b25dd3d34ce399,5ddc99d131adb502] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 20:59:43.876 INFO 16270 --- [ocp-schedules-8,078b52336a7fa4d3,06c92ff2d96951f5] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} inited 2025-04-08 20:59:43.880 INFO 16270 --- [ocp-schedules-2,88a17030b004577a,a539cb8356d3559c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T20:59:43.879+08:00, clusters=[2] 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : binlog.cluster.parameter.blacklist= 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/root/ocp-server/log/ocp.log 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/oceanbaselog/sgkyocp/ocp-server.log 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(16270){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 16270 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2025-04-08 20:59:43.881 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2025-04-08 20:59:43.882 INFO 16270 --- [ocp-schedules-22,798c68b1bcbf071f,4bd55688a6f3e570] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.force.ssh.installation=false 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80},{"file_regex":"[a-z]+.log.[0-9]+.zst","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.enabled=true 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.run.internal=5m 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.use.specific.http.ip=false 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full=4.3.5-20250319105844 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2025-04-08 20:59:43.882 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2025-04-08 20:59:43.883 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2025-04-08 20:59:43.883 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2025-04-08 20:59:43.883 INFO 16270 --- [ocp-schedules-10,67f18950844b2ef3,2f8cab483a487af9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.application.uuid=493eecb387654b6f90f5379d8459859c 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2025-04-08 20:59:43.883 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.object-storage-sdk.invoke.https=true 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2025-04-08 20:59:43.884 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2025-04-08 20:59:43.885 INFO 16270 --- [ocp-schedules-22,798c68b1bcbf071f,4bd55688a6f3e570] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2025-04-08 20:59:43.885 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/root/ocp/ocp-server/lib 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/root/software 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2025-04-08 20:59:43.886 INFO 16270 --- [ocp-schedules-3,3157d6a19ac05c3a,2bfa9bc16dbc65e1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T20:59:43.886+08:00, clusters=[2] 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2025-04-08 20:59:43.886 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/root/ocp/logs/ocp 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name=ocpdb 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2025-04-08 20:59:43.887 INFO 16270 --- [ocp-async-5,c63b286ed3836f6a,bf2c568c18d86e22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":3000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time=1744114820 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2025-04-08 20:59:43.887 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.rollup.type=MAX 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.service.type=LOCAL 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=192.168.1.61 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2881 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2025-04-08 20:59:43.888 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2025-04-08 20:59:43.889 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2025-04-08 20:59:43.889 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2025-04-08 20:59:43.893 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 20:59:43.893 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 20:59:43.893 INFO 16270 --- [ocp-async-4,7ffe87b0b2d4bc9a,43a8033c447deb80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 20:59:43.893 INFO 16270 --- [ocp-schedules-27,48f21698ae7500e7,480166753f9e7d11] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2025-04-08 20:59:43.895 INFO 16270 --- [ocp-schedules-25,b19254e5a5518516,fee04aa092419d81] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.batch.replace.arbitration.thread.core-pool-size=100 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist=ocpdb 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-zA-Z_0-9-]{2,64}$ 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2025-04-08 20:59:43.895 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2025-04-08 20:59:43.895 INFO 16270 --- [ocp-schedules-30,9500cac72050a2ac,a9b54890d4eb714d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 20:59:43.896 INFO 16270 --- [ocp-schedules-9,471c2ceaa246d398,5bf9cb5b6f89f00e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 20:59:43.896 INFO 16270 --- [ocp-schedules-30,9500cac72050a2ac,a9b54890d4eb714d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 20:59:43.896 INFO 16270 --- [ocp-schedules-18,351282384239d072,940a7263bc73edcc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 20:59:43.896 INFO 16270 --- [ocp-schedules-15,a33e5ec393c15f4e,512c60100805c77c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 20:59:43.896 INFO 16270 --- [ocp-schedules-14,f86cd6d79dc1f176,c91d4fec7c34bac7] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 20:59:43.907 INFO 16270 --- [ocp-schedules-19,2413e4d559ad6c20,43e10fad9c2dd7f8] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 20:59:43.907 WARN 16270 --- [ocp-async-7,8469805391db0ce7,acac07b39ef69cba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 20:59:43.908 INFO 16270 --- [ocp-schedules-28,168ab4e80c80d747,e58082774e7b487d] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 20:59:43.908 INFO 16270 --- [ocp-schedules-12,c6caed5054c6df04,ddcc6a2a4d54748a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 20:59:43.908 INFO 16270 --- [ocp-schedules-23,6b65c6c3689b59b9,fd7c60c567e57fda] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 20:59:43.909 WARN 16270 --- [ocp-schedules-23,6b65c6c3689b59b9,fd7c60c567e57fda] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=checkClusterCgroupValue, confName=checkClusterCgroupValue 2025-04-08 20:59:43.909 ERROR 16270 --- [ocp-schedules-23,6b65c6c3689b59b9,fd7c60c567e57fda] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.checkClusterCgroupValue(ClusterSchedules.java:135) ... 15 common frames omitted 2025-04-08 20:59:43.909 INFO 16270 --- [ocp-schedules-4,2d6783eba24e6ad1,2fabf5c373bf6a6a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 20:59:43.909 INFO 16270 --- [ocp-schedules-19,2413e4d559ad6c20,43e10fad9c2dd7f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:59:43.910 WARN 16270 --- [ocp-schedules-9,471c2ceaa246d398,5bf9cb5b6f89f00e] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncAllTenantInfo, confName=syncAllTenantInfo 2025-04-08 20:59:43.910 INFO 16270 --- [ocp-schedules-18,351282384239d072,940a7263bc73edcc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 20:59:43.910 ERROR 16270 --- [ocp-schedules-9,471c2ceaa246d398,5bf9cb5b6f89f00e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) ... 15 common frames omitted 2025-04-08 20:59:43.910 INFO 16270 --- [ocp-schedules-12,c6caed5054c6df04,ddcc6a2a4d54748a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:59:43.910 INFO 16270 --- [ocp-schedules-5,8b0708a4c765853f,b04892889d949436] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:59:43.911 INFO 16270 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 20:59:43.911 INFO 16270 --- [ocp-schedules-7,d2042ed4d4f8e909,c7b75962e9a85c15] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 20:59:43.911 INFO 16270 --- [ocp-schedules-15,a33e5ec393c15f4e,512c60100805c77c] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 20:59:43.911 INFO 16270 --- [ocp-schedules-14,f86cd6d79dc1f176,c91d4fec7c34bac7] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 20:59:43.914 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2025-04-08 20:59:43.914 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obshell.port=2886 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2025-04-08 20:59:43.915 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2025-04-08 20:59:43.916 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2025-04-08 20:59:43.917 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2025-04-08 20:59:43.918 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://192.168.1.61 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2025-04-08 20:59:43.919 INFO 16270 --- [ocp-schedules-4,2d6783eba24e6ad1,2fabf5c373bf6a6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2025-04-08 20:59:43.919 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.fixedDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.initialDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2025-04-08 20:59:43.922 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2025-04-08 20:59:43.928 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2025-04-08 20:59:43.929 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationAutoReplaceStatusTask.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationServiceGroup.fixedDelay=10 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterCgroupValue.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkSuspendTenantCompactionStatus.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2025-04-08 20:59:43.930 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.oracle=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.5 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.5-20250319105844 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2025-04-08 20:59:43.931 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8080 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2025-04-08 20:59:43.932 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2025-04-08 20:59:43.933 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2025-04-08 20:59:43.933 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2025-04-08 20:59:43.933 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2025-04-08 20:59:43.933 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2025-04-08 20:59:43.933 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2025-04-08 20:59:43.934 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2025-04-08 20:59:43.935 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2025-04-08 20:59:43.936 INFO 16270 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2025-04-08 20:59:43.944 INFO 16270 --- [ocp-schedules-1,4d9d469bc7930aa8,a6f04c710007992a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 20:59:43.951 INFO 16270 --- [ocp-schedules-2,88a17030b004577a,a539cb8356d3559c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 20:59:43.956 INFO 16270 --- [ocp-schedules-3,3157d6a19ac05c3a,2bfa9bc16dbc65e1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 20:59:43.959 INFO 16270 --- [pool-overview-container-8,202b833ca92172d9,bb77e56a6793dac1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:59:43.959 INFO 16270 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 103.958 seconds (JVM running for 105.575) 2025-04-08 20:59:43.914 INFO 16270 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 20:59:43.972 INFO 16270 --- [ocp-schedules-23,b34799e7b0a66ab3,ad83699358eebeff] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 20:59:43.972 INFO 16270 --- [ocp-schedules-29,9f4fbc1521cc5f87,fdda8a1875ba33ad] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 20:59:43.968 INFO 16270 --- [ocp-schedules-18,351282384239d072,940a7263bc73edcc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 20:59:43.972 INFO 16270 --- [ocp-schedules-18,351282384239d072,940a7263bc73edcc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 20:59:43.973 INFO 16270 --- [ocp-schedules-6,e58e0a414a5153d3,7a9672daf4f1641e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 20:59:43.975 INFO 16270 --- [pool-overview-container-1,de64119bdf513b5e,88e33ca8d895f9a0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:59:43.975 INFO 16270 --- [ocp-schedules-11,e3a2033865fd0cd9,0be050f27782bea0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 20:59:43.976 INFO 16270 --- [ocp-schedules-11,e3a2033865fd0cd9,0be050f27782bea0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 20:59:43.983 INFO 16270 --- [ocp-schedules-17,a90e4b66218469c2,617afa00d12fddf7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 20:59:43.984 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:59:43.984 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:59:43.984 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:59:43.984 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:59:43.986 INFO 16270 --- [ocp-schedules-27,48f21698ae7500e7,480166753f9e7d11] c.o.o.m.service.SeriesIdCleanService : HostIpSize=1, clusterSize=1, obproxyClusterSize=0 2025-04-08 20:59:43.987 INFO 16270 --- [ocp-schedules-28,168ab4e80c80d747,e58082774e7b487d] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 20:59:43.987 INFO 16270 --- [ocp-schedules-28,168ab4e80c80d747,e58082774e7b487d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 20:59:43.987 INFO 16270 --- [ocp-schedules-28,168ab4e80c80d747,e58082774e7b487d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 20:59:43.993 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 20:59:43.993 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 20:59:43.993 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 20:59:43.993 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 20:59:44.006 INFO 16270 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2025-04-08 20:59:44.011 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2025-04-08 20:59:44.012 INFO 16270 --- [Check status for all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_status_check', labels={app=OB, check_item=cluster connection, failed_reason=-, ob_cluster_id=1744114127, ob_cluster_name=ocpdb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744117183 : 2025-04-08 12:59:43 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2025-04-08 20:59:44.012 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2025-04-08 20:59:44.035 INFO 16270 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2025-04-08 20:59:44.036 INFO 16270 --- [ocp-schedules-8,078b52336a7fa4d3,06c92ff2d96951f5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 20:59:44.037 INFO 16270 --- [ocp-schedules-8,078b52336a7fa4d3,06c92ff2d96951f5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 20:59:44.041 INFO 16270 --- [ocp-schedules-13,152372a4c5966590,b63b7a5c3e02c09c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 20:59:44.048 INFO 16270 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 493eecb387654b6f90f5379d8459859c. 2025-04-08 20:59:44.048 INFO 16270 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2025-04-08 20:59:44.055 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1M44.049374394S 2025-04-08 20:59:44.055 INFO 16270 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2025-04-08 20:59:44.062 INFO 16270 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2025-04-08 20:59:44.062 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 20:59:44.062 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:59:44.063 INFO 16270 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2025-04-08 20:59:44.064 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 20:59:44.069 INFO 16270 --- [pool-21-thread-1,a33e5ec393c15f4e,1d1eb708ac97c190] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 20:59:44.080 INFO 16270 --- [ocp-schedules-27,48f21698ae7500e7,480166753f9e7d11] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2025-04-08 20:59:44.080 INFO 16270 --- [ocp-schedules-27,48f21698ae7500e7,480166753f9e7d11] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 20:59:44.080 INFO 16270 --- [ocp-schedules-27,48f21698ae7500e7,480166753f9e7d11] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 20:59:44.081 ERROR 16270 --- [ocp-schedules-4,2d6783eba24e6ad1,2fabf5c373bf6a6a] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$13fd79d4.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:59:44.081 INFO 16270 --- [ocp-async-7,f613403cc229343e,28e973cb43193950] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [2] 2025-04-08 20:59:44.082 ERROR 16270 --- [ocp-schedules-19,2413e4d559ad6c20,43e10fad9c2dd7f8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$13fd79d4.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 20:59:44.085 INFO 16270 --- [pool-11-thread-3,de64119bdf513b5e,1d152beb7ff93d44] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 20:59:44.086 INFO 16270 --- [ocp-schedules-29,9f4fbc1521cc5f87,fdda8a1875ba33ad] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 20:59:44.090 INFO 16270 --- [ocp-schedules-4,2d6783eba24e6ad1,2fabf5c373bf6a6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 20:59:44.090 INFO 16270 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:59:44.093 INFO 16270 --- [ocp-schedules-19,2413e4d559ad6c20,43e10fad9c2dd7f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 20:59:44.094 INFO 16270 --- [pool-11-thread-4,de64119bdf513b5e,9e24e061a1c738d0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 20:59:44.094 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:59:44.094 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:59:44.094 INFO 16270 --- [ForkJoinPool-7-worker-9,de64119bdf513b5e,d46ddb17e77192d7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 20:59:44.094 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:59:44.094 INFO 16270 --- [ForkJoinPool-7-worker-9,de64119bdf513b5e,d46ddb17e77192d7] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 20:59:44.094 INFO 16270 --- [ForkJoinPool-7-worker-9,de64119bdf513b5e,d46ddb17e77192d7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 20:59:44.094 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:59:44.094 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:59:44.096 INFO 16270 --- [pool-11-thread-2,de64119bdf513b5e,04ed3eb13ba71ce4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 20:59:44.100 WARN 16270 --- [ocp-async-7,f613403cc229343e,28e973cb43193950] c.o.o.p.a.i.SessionEventSyncService : Fail to refresh cluster event info, cluster id = 2 2025-04-08 20:59:44.100 INFO 16270 --- [ocp-async-7,f613403cc229343e,28e973cb43193950] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [2] 2025-04-08 20:59:44.101 INFO 16270 --- [ForkJoinPool-7-worker-2,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 20:59:44.102 INFO 16270 --- [pool-11-thread-1,de64119bdf513b5e,cda801cd154306d5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 20:59:44.102 INFO 16270 --- [pool-11-thread-1,de64119bdf513b5e,cda801cd154306d5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 20:59:44.103 INFO 16270 --- [ForkJoinPool-7-worker-2,de64119bdf513b5e,4f208b9127390fe5] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 20:59:44.106 INFO 16270 --- [pool-11-thread-3,de64119bdf513b5e,1d152beb7ff93d44] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 20:59:44.108 INFO 16270 --- [pool-11-thread-4,de64119bdf513b5e,9e24e061a1c738d0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 20:59:44.110 WARN 16270 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2025-04-08 20:59:44.110 INFO 16270 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2025-04-08 20:59:44.110 INFO 16270 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2025-04-08 20:59:44.111 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 20:59:44.114 ERROR 16270 --- [ocp-async-8,62ec949770ee05e8,1dd54883cd15313f] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2025-04-08 20:59:44.114 INFO 16270 --- [ocp-async-8,62ec949770ee05e8,1dd54883cd15313f] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2025-04-08 20:59:44.117 WARN 16270 --- [ocp-schedules-15,a33e5ec393c15f4e,512c60100805c77c] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$13fd79d4.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$71eff128.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 20:59:44.117 INFO 16270 --- [ocp-schedules-15,a33e5ec393c15f4e,512c60100805c77c] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 20:59:44.118 INFO 16270 --- [ocp-async-8,62ec949770ee05e8,1dd54883cd15313f] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2025-04-08 20:59:44.122 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=423, maxSyncedRecordId=766200877631795200 2025-04-08 20:59:44.123 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 20:59:44.123 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 20:59:44.126 INFO 16270 --- [ocp-async-8,62ec949770ee05e8,1dd54883cd15313f] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2025-04-08 20:59:44.129 INFO 16270 --- [pool-11-thread-1,de64119bdf513b5e,cda801cd154306d5] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 20:59:44.132 INFO 16270 --- [pool-11-thread-2,de64119bdf513b5e,04ed3eb13ba71ce4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 20:59:44.137 INFO 16270 --- [pool-11-thread-1,de64119bdf513b5e,cda801cd154306d5] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 20:59:44.143 INFO 16270 --- [pool-11-thread-2,de64119bdf513b5e,04ed3eb13ba71ce4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 20:59:44.163 INFO 16270 --- [ocp-schedules-24,198aaa3ed6d2f71c,5f7be3f6ef603565] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=266, maxSyncedRecordId=766198483195916288 2025-04-08 20:59:44.405 INFO 16270 --- [ocp-async-10,8c134d711b7b9e2b,16493d2961a718ed] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2025-04-08 20:59:44.445 INFO 16270 --- [ocp-schedules-10,67f18950844b2ef3,2f8cab483a487af9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:59:44.448 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=ocpdb, obClusterId=1744114127, tenantName=sys, username=root, address=192.168.1.61, port=2881, database=oceanbase) 2025-04-08 20:59:44.448 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 20:59:44.461 WARN 16270 --- [ocp-schedules-10,67f18950844b2ef3,2f8cab483a487af9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:59:44.463 INFO 16270 --- [ocp-schedules-10,67f18950844b2ef3,2f8cab483a487af9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 20:59:44.465 INFO 16270 --- [obsdk-init-thread-1,8db051dd2af7f821,7fa83e479841f3a0] com.alibaba.druid.pool.DruidDataSource : {dataSource-3,ob-connector-datasource_direct_sys_root_192.168.1.61_2881} inited 2025-04-08 20:59:44.465 INFO 16270 --- [obsdk-init-thread-1,8db051dd2af7f821,7fa83e479841f3a0] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 20:59:44.465 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 20:59:44.467 WARN 16270 --- [ocp-schedules-10,67f18950844b2ef3,2f8cab483a487af9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 20:59:44.478 INFO 16270 --- [ocp-schedules-10,67f18950844b2ef3,2f8cab483a487af9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=1 2025-04-08 20:59:44.488 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:59:44.489 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 20:59:44.513 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 20:59:44.526 INFO 16270 --- [ocp-async-10,8c134d711b7b9e2b,16493d2961a718ed] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=623 2025-04-08 20:59:44.526 INFO 16270 --- [ocp-async-10,8c134d711b7b9e2b,16493d2961a718ed] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2025-04-08 20:59:44.538 INFO 16270 --- [ocp-schedules-16,1a1dde21c10223f0,5e0d035f20f686dd] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 20:59:44.730 INFO 16270 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 20:59:44.734 INFO 16270 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 20:59:44.737 INFO 16270 --- [ocp-async-3,8db051dd2af7f821,fa3488d32bb00740] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 20:59:58.501 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 20:59:58.501 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 20:59:58.501 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 20:59:58.501 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 20:59:58.501 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 20:59:58.502 INFO 21094 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2025-04-08 20:59:58.510 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:00:04.037 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 21:00:04.037 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 21:00:05.500 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1442 ms. Found 173 JPA repository interfaces. 2025-04-08 21:00:05.507 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 21:00:05.507 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 21:00:05.683 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 176 ms. Found 16 JPA repository interfaces. 2025-04-08 21:00:05.683 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 21:00:05.683 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 21:00:05.731 INFO 21094 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 47 ms. Found 5 JPA repository interfaces. 2025-04-08 21:00:08.562 INFO 21094 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=535a2406-c3b5-3b8b-8149-b06173f75dfb 2025-04-08 21:00:09.249 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#59e082f8' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.509 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#59e082f8' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.550 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2bdb5e0f' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.564 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6cceb281' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.575 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6cceb281' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.600 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration' of type [org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.614 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metadbConfiguration' of type [com.oceanbase.ocp.server.common.config.MetadbConfiguration$$EnhancerBySpringCGLIB$$4dd0e38c] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.644 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration' of type [org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.653 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.660 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'loadBalancerClientsDefaultsMappingsProvider' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration$$Lambda$663/625411191] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.699 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultsBindHandlerAdvisor' of type [org.springframework.cloud.commons.config.DefaultsBindHandlerAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.720 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpClusterProperty.MetaDbProperties' of type [com.oceanbase.ocp.core.config.OcpClusterProperty$MetaDbProperties$$EnhancerBySpringCGLIB$$e615319b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.738 INFO 21094 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 21:00:09.738 INFO 21094 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2025-04-08 21:00:09.738 INFO 21094 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 21:00:09.744 INFO 21094 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2025-04-08 21:00:09.803 INFO 21094 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2025-04-08 21:00:09.811 INFO 21094 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 21:00:09.811 INFO 21094 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 21:00:09.812 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaDataSource' of type [com.zaxxer.hikari.HikariDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.843 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.sql.init-org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties' of type [org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.869 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceScriptDatabaseInitializer' of type [org.springframework.boot.autoconfigure.sql.init.SqlDataSourceScriptDatabaseInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.896 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.908 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa.hibernate-org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.925 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.938 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hikariPoolDataSourceMetadataProvider' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration$$Lambda$431/2123269713] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:09.996 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:10.043 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaVendorAdapter' of type [org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:10.065 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'entityManagerFactoryBuilder' of type [org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:10.248 INFO 21094 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 21:00:10.572 INFO 21094 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2025-04-08 21:00:11.347 INFO 21094 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2025-04-08 21:00:14.316 INFO 21094 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 21:00:15.645 WARN 21094 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2025-04-08 21:00:16.537 WARN 21094 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 21:00:16.537 WARN 21094 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 21:00:19.965 INFO 21094 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 21:00:20.043 INFO 21094 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 21:00:20.062 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.078 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [com.sun.proxy.$Proxy294] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.188 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3335afcf' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.387 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'propertyManager.MonitorDbExtraProperties' of type [com.oceanbase.ocp.core.property.PropertyManager$MonitorDbExtraProperties$$EnhancerBySpringCGLIB$$a2cbb52e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.392 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'monitordbConfiguration' of type [com.oceanbase.ocp.server.common.config.MonitordbConfiguration$$EnhancerBySpringCGLIB$$8b9fe823] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.471 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpAzoneModelConverter' of type [com.oceanbase.ocp.ha.maz.OcpAzoneModelConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.494 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1b904438' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.494 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1b904438' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.495 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#710e1af6' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.496 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#210bdb8e' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.496 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#210bdb8e' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.498 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#20520eb6' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.546 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.repository.config.JpaMetamodelMappingContextFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:20.554 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.mapping.JpaMetamodelMappingContext] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.491 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.505 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [com.sun.proxy.$Proxy300] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.562 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#673b5922' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.562 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#673b5922' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.563 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5b6626e5' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.564 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e96ff9a' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.564 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e96ff9a' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.566 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#518b5d87' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.919 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.920 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [com.sun.proxy.$Proxy304] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.941 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27b8eecb' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.941 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27b8eecb' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.942 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#292a73f2' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.942 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#71d8985' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.943 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#71d8985' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.944 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5f12dd3d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.972 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.972 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [com.sun.proxy.$Proxy305] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.994 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5b86ab5f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.994 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5b86ab5f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.995 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#78379715' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.996 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#68fab818' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.996 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#68fab818' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:21.997 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#22d989c0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.036 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.037 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [com.sun.proxy.$Proxy306] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.077 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2568c451' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.078 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2568c451' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.078 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7fb3a63c' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.079 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74973085' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.079 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74973085' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.081 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1352cd6a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.109 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.110 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [com.sun.proxy.$Proxy307] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.180 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaTransactionManager' of type [org.springframework.orm.jpa.JpaTransactionManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.478 INFO 21094 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action group type info: {"HOST_MANAGE":"HOST","ALL":"UNKNOWN","HOST_MAINTAIN":"HOST","ALL_READ":"UNKNOWN","OBPROXY_MAINTAIN":"OBPROXY","ORGANIZATION_READ":"ORGANIZATION","HOST_CREATE":"HOST","USER_MANAGE":"USER","ARBITRATION_READ":"ARBITRATION","ARBITRATION_CREATE":"ARBITRATION","BINLOG_MANAGE":"BINLOG","OBPROXY_MANAGE":"OBPROXY","HOST_READ":"HOST","BINLOG_READ":"BINLOG","AUDIT_READ":"AUDIT","USER_READ":"USER","ROLE_READ":"ROLE","CLUSTER_MANAGE":"CLUSTER","CLUSTER_CREATE":"CLUSTER","TENANT_MANAGE":"TENANT","CLUSTER_MAINTAIN":"CLUSTER","TENANT_MAINTAIN":"TENANT","CLUSTER_READ":"CLUSTER","BINLOG_MAINTAIN":"BINLOG","TENANT_READ":"TENANT","OBPROXY_READ":"OBPROXY","ARBITRATION_MAINTAIN":"ARBITRATION","ROLE_MANAGE":"ROLE","BINLOG_CREATE":"BINLOG","ARBITRATION_MANAGE":"ARBITRATION","OBPROXY_CREATE":"OBPROXY"} 2025-04-08 21:00:22.480 INFO 21094 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action type info: {"ALARM:CREATE":"ALARM","SELFCURE:UPDATE":"SELFCURE","HOST:DELETE":"HOST","ROLE:CREATE":"ROLE","HOST:UPDATE":"HOST","CLUSTER:UPDATE":"CLUSTER","INSPECTION:UPDATE":"INSPECTION","CLUSTER:READ":"CLUSTER","PACKAGE:READ":"PACKAGE","ALARM:READ":"ALARM","TAG:DELETE":"TAG","BINLOG:CREATE":"BINLOG","ARBITRATION:CREATE":"ARBITRATION","SELFCURE:READ":"SELFCURE","*":"UNKNOWN","CLUSTER:DELETE":"CLUSTER","CREDENTIAL:DELETE":"CREDENTIAL","OBPROXY:READ":"OBPROXY","PROPERTY:READ":"PROPERTY","ARBITRATION:UPDATE":"ARBITRATION","OBPROXY:DELETE":"OBPROXY","TENANT:UPDATE":"TENANT","BINLOG:READ":"BINLOG","HOST:READ":"HOST","INSPECTION:CREATE":"INSPECTION","ALARM:UPDATE":"ALARM","PROPERTY:UPDATE":"PROPERTY","BINLOG:UPDATE":"BINLOG","PACKAGE:UPDATE":"PACKAGE","HOST:CREATE":"HOST","ROLE:DELETE":"ROLE","TAG:READ":"TAG","ROLE:READ":"ROLE","BINLOG:DELETE":"BINLOG","TASK:UPDATE":"TASK","USER:DELETE":"USER","CREDENTIAL:UPDATE":"CREDENTIAL","USER:UPDATE":"USER","TENANT:READ":"TENANT","TAG:UPDATE":"TAG","OBPROXY:CREATE":"OBPROXY","ARBITRATION:READ":"ARBITRATION","ROLE:UPDATE":"ROLE","ORGANIZATION:READ":"ORGANIZATION","TENANT:DELETE":"TENANT","USER:READ":"USER","AUDIT:READ":"AUDIT","TASK:READ":"TASK","TAG:CREATE":"TAG","INSPECTION:READ":"INSPECTION","ARBITRATION:DELETE":"ARBITRATION","USER:CREATE":"USER","PACKAGE:DELETE":"PACKAGE","ALARM:DELETE":"ALARM","CREDENTIAL:READ":"CREDENTIAL","OBPROXY:UPDATE":"OBPROXY","CREDENTIAL:CREATE":"CREDENTIAL","CLUSTER:CREATE":"CLUSTER"} 2025-04-08 21:00:22.480 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionMetaHelper' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ActionMetaHelper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.503 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultPerfConfiguration' of type [com.oceanbase.ocp.server.common.config.DefaultPerfConfiguration$$EnhancerBySpringCGLIB$$2af43631] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.537 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'authenticationFacadeImpl' of type [com.oceanbase.ocp.core.security.AuthenticationFacadeImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.560 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#31e1a699' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.561 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#31e1a699' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.562 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c13e774' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.562 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1f82267e' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.563 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1f82267e' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.564 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f308a2f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.640 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.641 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [com.sun.proxy.$Proxy310] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.669 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOperator' of type [com.oceanbase.ocp.core.security.organization.operator.HostOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.686 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#225620f4' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.687 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#225620f4' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.687 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f32afbe' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.688 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#168229e3' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.688 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#168229e3' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.690 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#14d4fe82' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.746 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.747 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [com.sun.proxy.$Proxy312] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.761 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObClusterOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.779 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d9803b4' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.779 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d9803b4' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.780 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4dfc46b4' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.781 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#52af8366' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.781 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#52af8366' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.782 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5a6a293d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.842 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.842 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [com.sun.proxy.$Proxy313] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.861 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObproxyOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.877 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5207c8ca' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.877 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5207c8ca' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.878 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#64ac6960' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.879 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74fd6fed' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.879 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74fd6fed' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.880 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#119940b4' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.914 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.915 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [com.sun.proxy.$Proxy315] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.929 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObTenantOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.945 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3c4b9a8f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.946 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3c4b9a8f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.946 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#23676bc' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.947 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#553f850d' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.947 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#553f850d' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:22.949 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#742b3192' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.008 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.009 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [com.sun.proxy.$Proxy316] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.027 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOperator' of type [com.oceanbase.ocp.core.security.organization.operator.UserOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.043 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3425709a' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.044 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3425709a' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.044 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#20337a79' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.045 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27241127' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.045 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#27241127' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.046 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#379ebfa0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.070 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.071 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [com.sun.proxy.$Proxy318] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.091 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ArbitrationOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.108 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4517959f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.109 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4517959f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.109 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#15f4dffe' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.110 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#356b37dd' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.110 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#356b37dd' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.111 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#d7ad9e5' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.145 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.146 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [com.sun.proxy.$Proxy319] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.158 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogOperator' of type [com.oceanbase.ocp.core.security.organization.operator.BinlogOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.160 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceManager' of type [com.oceanbase.ocp.core.security.organization.ResourceManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.174 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d3c799e' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.174 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d3c799e' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.175 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7ede650b' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.176 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#e36bbb9' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.176 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#e36bbb9' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.177 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#53508d4b' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.217 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.218 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [com.sun.proxy.$Proxy320] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.245 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b31c0ba' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.245 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3b31c0ba' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.246 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5c1c44b9' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.246 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1bad6995' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.246 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1bad6995' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.248 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#b7a9093' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.266 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.267 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [com.sun.proxy.$Proxy322] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.294 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43c6f461' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.294 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43c6f461' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.295 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#14400bf1' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.296 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6bd9d056' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.296 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6bd9d056' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.298 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#15eb82fc' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.326 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.327 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [com.sun.proxy.$Proxy323] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.344 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'getOrganizationService' of type [com.oceanbase.ocp.core.security.organization.OrganizationServiceImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.370 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#524206cb' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.371 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#524206cb' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.372 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#19d27788' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.372 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4b7cc75b' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.372 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4b7cc75b' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.373 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#79779c55' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.477 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.478 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [com.sun.proxy.$Proxy324] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.506 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#161267b6' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.507 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#161267b6' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.507 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4b6a81ab' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.508 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7e5ed9ce' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.508 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7e5ed9ce' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.509 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#35dbed4f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.540 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.540 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [com.sun.proxy.$Proxy325] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.565 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#50406145' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.565 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#50406145' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.566 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1561e62e' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.566 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#67dab5de' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.567 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#67dab5de' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.568 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#10ce2a4d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.627 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.627 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [com.sun.proxy.$Proxy326] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.657 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b7cd454' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.657 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b7cd454' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.658 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5f892e12' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.658 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f044be0' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.658 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f044be0' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.660 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#93568ed' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.689 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.690 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [com.sun.proxy.$Proxy327] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.711 INFO 21094 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:00:23.832 INFO 21094 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:00:23.832 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.853 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5951d91' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.853 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5951d91' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.854 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7bd44868' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.854 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1ac864d7' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.854 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1ac864d7' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.855 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2388f71' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.901 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.901 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [com.sun.proxy.$Proxy330] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.929 INFO 21094 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:00:23.941 INFO 21094 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:00:23.941 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obProxyQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObProxyQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.960 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4ae707c' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.960 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4ae707c' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.961 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2f8892c5' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.961 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c7108ce' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.962 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c7108ce' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.963 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#77b3a4c0' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.983 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:23.983 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [com.sun.proxy.$Proxy331] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.008 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#24766217' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.009 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#24766217' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.009 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5a42eadc' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.010 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7ea887ee' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.010 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7ea887ee' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.011 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1563dacc' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.048 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.049 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [com.sun.proxy.$Proxy332] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.069 INFO 21094 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:00:24.088 INFO 21094 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:00:24.088 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogQueryCache' of type [com.oceanbase.ocp.core.cache.helper.BinlogQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.091 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceCache' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ResourceCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.092 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.103 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'adminResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.113 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'clusterResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.122 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.132 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.142 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.151 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'tenantResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.153 INFO 21094 --- [main,,] c.o.o.c.s.a.r.ResourceSupplierManager : Resource supplier list: [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl@21c1fb2e, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl@39a95375, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl@7082d3e2, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl@291be813, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl@6cf14dc3, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl@5d0b5f31, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl@433b7810] 2025-04-08 21:00:24.153 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceSupplierManager' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.ResourceSupplierManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.179 INFO 21094 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$59e0b2b9] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:00:24.194 INFO 21094 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2025-04-08 21:00:24.659 INFO 21094 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=192.168.1.61:8080, name=b88d8e1c7d 2025-04-08 21:00:24.681 WARN 21094 --- [main,,] c.o.o.c.d.server.DefaultServerUpdater : Deleted expired server: DistributedServer{id=192.168.1.61:8080, name=b88d8e1c7d, expireInSeconds=30, createTime=2025-04-08T20:58:39+08:00, updateTime=2025-04-08T20:59:34+08:00, expireTime=2025-04-08T21:00:04+08:00, isActive=false} 2025-04-08 21:00:24.709 INFO 21094 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 192.168.1.61:8080 2025-04-08 21:00:25.536 INFO 21094 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2025-04-08 21:00:25.566 INFO 21094 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2025-04-08 21:00:25.566 INFO 21094 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.98] 2025-04-08 21:00:25.733 INFO 21094 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2025-04-08 21:00:25.733 INFO 21094 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 27223 ms 2025-04-08 21:00:26.108 INFO 21094 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2025-04-08 21:00:26.353 INFO 21094 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2025-04-08 21:00:26.353 INFO 21094 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2025-04-08 21:00:26.353 INFO 21094 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 21:00:26.383 INFO 21094 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 21:00:26.383 INFO 21094 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 21:00:26.873 INFO 21094 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2025-04-08 21:00:26.883 INFO 21094 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : host 192.168.1.61, port 2881, database monitor_database 2025-04-08 21:00:26.884 INFO 21094 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 21:00:26.884 INFO 21094 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2025-04-08 21:00:26.884 INFO 21094 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2025-04-08 21:00:26.884 INFO 21094 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2025-04-08 21:00:26.888 INFO 21094 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2025-04-08 21:00:26.888 INFO 21094 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 21:00:26.888 INFO 21094 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 21:00:27.490 INFO 21094 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 21:00:29.717 INFO 21094 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 21:00:30.090 INFO 21094 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 21:00:30.090 INFO 21094 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 21:00:44.128 INFO 21094 --- [main,,] org.reflections.Reflections : Reflections took 5441 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 21:00:49.149 INFO 21094 --- [main,,] org.reflections.Reflections : Reflections took 4959 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 21:00:53.519 INFO 21094 --- [main,,] org.reflections.Reflections : Reflections took 4357 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 21:00:53.533 INFO 21094 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2025-04-08 21:00:53.707 INFO 21094 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2025-04-08 21:00:54.217 INFO 21094 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2025-04-08 21:00:54.807 INFO 21094 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2025-04-08 21:00:54.829 INFO 21094 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2025-04-08 21:00:54.829 INFO 21094 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2025-04-08 21:00:55.658 INFO 21094 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=arbitration_group_auto_replace_failed,ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,arbitration_auto_replace_operation_info,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_merge_error,ob_tenant_name_mismatch, 2025-04-08 21:00:56.629 INFO 21094 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table switch time is 1744114820 2025-04-08 21:00:57.847 INFO 21094 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:00:57.880 INFO 21094 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:00:57.890 INFO 21094 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2025-04-08 21:00:58.858 INFO 21094 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:00:58.563+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion 2025-04-08 21:00:58.864 INFO 21094 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:00:58.864 INFO 21094 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:00:58.563+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion 2025-04-08 21:00:58.864 INFO 21094 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:00:58.865 INFO 21094 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:00:59.366 INFO 21094 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/root/ocp-server/alarm 2025-04-08 21:00:59.535 INFO 21094 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2025-04-08 21:00:59.586 INFO 21094 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=192.168.1.61:21094 2025-04-08 21:00:59.665 INFO 21094 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 21:00:59.785 INFO 21094 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 21:00:59.803 INFO 21094 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2025-04-08 21:00:59.844 INFO 21094 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2025-04-08 21:01:00.140 INFO 21094 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=1, heartbeatTime=2025-04-08T20:59:45+08:00, now=2025-04-08T21:01:00.139+08:00 2025-04-08 21:01:00.140 INFO 21094 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=2, heartbeatTime=2025-04-08T20:59:45+08:00, now=2025-04-08T21:01:00.139+08:00 2025-04-08 21:01:00.164 INFO 21094 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2025-04-08 21:01:00.168 INFO 21094 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2025-04-08 21:01:00.300 INFO 21094 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=192.168.1.61:8080, consumerId=2, resources=[ShardingSlice(consumerId=2, type=DEFAULT, resourceIds=[18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 80, 81, 106, 107, 250, 251, 252, 253, 254])] 2025-04-08 21:01:00.300 INFO 21094 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-target, name=192.168.1.61:8080, consumerId=1, resources=[ShardingSlice(consumerId=1, type=OBCluster, resourceIds=[2]), ShardingSlice(consumerId=1, type=Host, resourceIds=[2])] 2025-04-08 21:01:00.571 INFO 21094 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2025-04-08 21:01:01.153 INFO 21094 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2025-04-08 21:01:01.189 INFO 21094 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2025-04-08 21:01:01.413 INFO 21094 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/root/software 2025-04-08 21:01:03.302 INFO 21094 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from OFFLINE to NEW. 2025-04-08 21:01:03.980 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2025-04-08 21:01:03.981 INFO 21094 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2025-04-08 21:01:03.982 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2025-04-08 21:01:03.998 INFO 21094 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2025-04-08 21:01:04.000 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2025-04-08 21:01:04.000 INFO 21094 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2025-04-08 21:01:04.001 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2025-04-08 21:01:04.001 INFO 21094 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2025-04-08 21:01:04.003 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2025-04-08 21:01:04.003 INFO 21094 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2025-04-08 21:01:04.004 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2025-04-08 21:01:04.004 INFO 21094 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2025-04-08 21:01:04.006 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2025-04-08 21:01:04.006 INFO 21094 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2025-04-08 21:01:04.007 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2025-04-08 21:01:04.009 INFO 21094 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2025-04-08 21:01:04.011 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2025-04-08 21:01:04.011 INFO 21094 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2025-04-08 21:01:04.012 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2025-04-08 21:01:04.012 INFO 21094 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2025-04-08 21:01:04.013 INFO 21094 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2025-04-08 21:01:04.013 INFO 21094 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2025-04-08 21:01:05.302 INFO 21094 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=1, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@4d4e1369 2025-04-08 21:01:05.304 INFO 21094 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=2, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@133d23d8 2025-04-08 21:01:05.305 INFO 21094 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 21:01:05.305 INFO 21094 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 21:01:06.355 INFO 21094 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 21:01:06.363 INFO 21094 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 21:01:06.948 INFO 21094 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 21:01:06.957 INFO 21094 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 21:01:09.382 INFO 21094 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 21:01:09.382 INFO 21094 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 21:01:09.388 INFO 21094 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 21:01:09.388 INFO 21094 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 21:01:10.313 INFO 21094 --- [main,,] org.reflections.Reflections : Reflections took 4289 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 21:01:10.325 INFO 21094 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2025-04-08 21:01:10.326 INFO 21094 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2025-04-08 21:01:10.326 INFO 21094 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2025-04-08 21:01:10.327 INFO 21094 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2025-04-08 21:01:10.327 INFO 21094 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2025-04-08 21:01:10.344 INFO 21094 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on b88d8e1c7d 2025-04-08 21:01:11.351 INFO 21094 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2025-04-08 21:01:12.016 INFO 21094 --- [main,,] c.o.o.s.g.prefs.PrefsParameterService : Start loading prefs template. 2025-04-08 21:01:12.446 INFO 21094 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Begin init binlog cluster parameter info. 2025-04-08 21:01:12.457 INFO 21094 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Finish init binlog cluster parameter info, parameter blacklist: [] 2025-04-08 21:01:12.508 INFO 21094 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Begin init binlog instance parameter info. 2025-04-08 21:01:12.514 INFO 21094 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Finish init binlog instance parameter info, parameter blacklist: [] 2025-04-08 21:01:13.429 INFO 21094 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2025-04-08 21:01:14.490 INFO 21094 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2025-04-08 21:01:14.522 INFO 21094 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB, valid=true 2025-04-08 21:01:14.522 INFO 21094 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Sensitive data encryption key is legal. 2025-04-08 21:01:14.629 INFO 21094 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2025-04-08 21:01:14.898 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 21:01:14.934 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 21:01:14.955 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:01:14.964 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:01:16.303 INFO 21094 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 192.168.1.61:8080 2025-04-08 21:01:16.313 INFO 21094 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2025-04-08 21:01:16.360 WARN 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 21:01:16.360 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 21:01:16.741 INFO 21094 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2025-04-08 21:01:16.754 INFO 21094 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2025-04-08 21:01:18.323 INFO 21094 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/root/ocp/ocp-server/lib 2025-04-08 21:01:18.323 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2025-04-08 21:01:18.352 INFO 21094 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=obproxyd.sh, objectId=1000001 2025-04-08 21:01:18.370 INFO 21094 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2025-04-08 21:01:18.398 INFO 21094 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh.sha1sum 2025-04-08 21:01:18.398 INFO 21094 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh 2025-04-08 21:01:18.398 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2025-04-08 21:01:18.454 INFO 21094 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=1000002 2025-04-08 21:01:18.492 INFO 21094 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=1000002, name=obproxyd.sh 2025-04-08 21:01:18.505 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 21:01:18.526 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 21:01:18.526 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 21:01:18.529 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 21:01:18.529 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:01:18.532 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:01:18.532 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:01:18.536 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:01:18.536 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 21:01:18.539 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 21:01:18.540 INFO 21094 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.5-20250319105844 2025-04-08 21:01:18.546 INFO 21094 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.5-20250319105844, updatedValue=4.3.5-20250319105844, runningValue=4.3.5-20250319105844 2025-04-08 21:01:18.590 INFO 21094 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"update_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition","timeoutNs":0},"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event","timeoutNs":0},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception","timeoutNs":0},"cleanStorageObjectBlock":{"batchIntervalMillis":10000,"batchSize":10,"condition":"file_id not in (select id from storage_object_meta)","priority":0,"tableName":"storage_object_block","timeoutNs":0}} 2025-04-08 21:01:19.027 INFO 21094 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2025-04-08 21:01:19.278 INFO 21094 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2025-04-08 21:01:19.280 INFO 21094 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2025-04-08 21:01:19.361 WARN 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 21:01:19.361 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 21:01:20.581 INFO 21094 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2025-04-08 21:01:20.686 INFO 21094 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2025-04-08 21:01:20.687 INFO 21094 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117320, interval=60 2025-04-08 21:01:20.689 INFO 21094 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2025-04-08 21:01:20.726 INFO 21094 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2025-04-08 21:01:20.975 INFO 21094 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2025-04-08 21:01:21.270 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 21:01:21.278 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 21:01:21.281 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:01:21.282 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:01:21.282 INFO 21094 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2025-04-08 21:01:21.296 INFO 21094 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2025-04-08 21:01:21.301 INFO 21094 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2025-04-08 21:01:21.307 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=CMDB_SYNCER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:44+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.308 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=MC_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:44+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.310 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PERF_SQL_ROLLUP, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:34+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.311 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:34+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.313 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=RATE_LIMIT, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:44+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.314 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:34+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.316 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:34+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.321 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=TASK_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T20:59:34+08:00, updateTime=2025-04-08T20:59:44+08:00, expireTime=null, isLeader=false} 2025-04-08 21:01:21.322 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.327 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.329 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.350 INFO 21094 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2025-04-08 21:01:21.350 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.356 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.363 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.365 INFO 21094 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2025-04-08 21:01:21.367 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.384 WARN 21094 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 21:01:21.740 INFO 21094 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2025-04-08 21:01:21.775 INFO 21094 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=2, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@49763e08 2025-04-08 21:01:21.775 INFO 21094 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=192.168.1.61:8080, consumerId=3, resources=[ShardingSlice(consumerId=3, type=OB_CLUSTER, resourceIds=[2]), ShardingSlice(consumerId=3, type=HOST_MONITOR, resourceIds=[2])] 2025-04-08 21:01:21.775 INFO 21094 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 21:01:22.361 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 gets the leadership... 2025-04-08 21:01:22.426 INFO 21094 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2025-04-08 21:01:22.440 INFO 21094 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2025-04-08 21:01:22.442 INFO 21094 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2025-04-08 21:01:22.445 INFO 21094 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2025-04-08 21:01:22.774 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$1994/182412825@7ce90ee6 with [org.springframework.security.web.session.DisableEncodeUrlFilter@7b9d7046, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@3636de1d, org.springframework.security.web.context.SecurityContextPersistenceFilter@3edb4a6c, org.springframework.security.web.header.HeaderWriterFilter@4aa71f42, org.springframework.security.web.authentication.logout.LogoutFilter@7d24f994, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@52ace054, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@66065ad9, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@7689191f, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@76cfe7e5, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@71953583, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@540e74dc, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@357ca580, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@43d41728, org.springframework.security.web.session.SessionManagementFilter@7ab997df, org.springframework.security.web.access.ExceptionTranslationFilter@538498ef, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@4c591b5f] 2025-04-08 21:01:22.785 INFO 21094 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2025-04-08 21:01:22.785 INFO 21094 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2025-04-08 21:01:22.989 INFO 21094 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2025-04-08 21:01:23.130 INFO 21094 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2025-04-08 21:01:23.136 INFO 21094 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2025-04-08 21:01:23.150 INFO 21094 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2025-04-08 21:01:24.291 INFO 21094 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2025-04-08 21:01:24.292 INFO 21094 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2025-04-08 21:01:24.606 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@66ed8a54, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@30d570f1, org.springframework.security.web.context.SecurityContextPersistenceFilter@75b4ecf9, org.springframework.security.web.header.HeaderWriterFilter@3216b669, org.springframework.security.web.csrf.CsrfFilter@7cddc0c1, org.springframework.security.web.authentication.logout.LogoutFilter@394062e7, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@29b2e8be, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@512d0e7c, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@62b056cb, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@779d5261, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@3ca2fb7, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@45fa64fe, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@4fcff93f, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@4eebc5db, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@31f8a6b0, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@365aeb18, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@54064acf, org.springframework.security.web.session.SessionManagementFilter@205fb322, org.springframework.security.web.access.ExceptionTranslationFilter@e295433, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@4c17362a] 2025-04-08 21:01:24.643 INFO 21094 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2025-04-08 21:01:24.776 INFO 21094 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 21:01:24.811 INFO 21094 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 21:01:25.072 INFO 21094 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2025-04-08 21:01:25.361 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:01:25.368 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:01:25.394 WARN 21094 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2025-04-08 21:01:26.286 INFO 21094 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2025-04-08 21:01:27.794 INFO 21094 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 21:01:27.794 INFO 21094 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2025-04-08 21:01:28.369 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:01:28.370 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/static/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/static/**'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff2']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff2'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.ttf']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.ttf'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.tif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.tif'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.gif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.gif'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.jpg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.jpg'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.png']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.png'] 2025-04-08 21:01:29.529 WARN 21094 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:01:29.529 INFO 21094 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2025-04-08 21:01:30.310 INFO 21094 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2025-04-08 21:01:30.604 INFO 21094 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2025-04-08 21:01:30.836 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=15 2025-04-08 21:01:30.836 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=inhibitAlarms, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=6 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.837 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkSuspendTenantCompactionStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=11 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterCgroupValue, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationServiceGroup, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.838 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationAutoReplaceStatusTask, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.856 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2025-04-08 21:01:30.856 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2025-04-08 21:01:30.856 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.862 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.862 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.862 INFO 21094 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:01:30.875 INFO 21094 --- [ocp-schedules-20,98ccb232dfd33c58,52f62e7946ba2f41] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:01:30.875 INFO 21094 --- [ocp-schedules-21,11427c0d51ec81dc,3cd20f11587224f0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:01:30.877 INFO 21094 --- [ocp-schedules-22,5ce6fa89333fb78c,e7e947244016d49b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:01:30.878 WARN 21094 --- [ocp-schedules-21,11427c0d51ec81dc,3cd20f11587224f0] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncClusterInfo, confName=syncClusterInfo 2025-04-08 21:01:30.880 INFO 21094 --- [ocp-schedules-22,5ce6fa89333fb78c,e7e947244016d49b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:01:30.881 INFO 21094 --- [ocp-schedules-1,c0cdf11548436b2c,2de7a64efe0bf80f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:01:30.880+08:00, clusters=[2] 2025-04-08 21:01:30.882 INFO 21094 --- [ocp-schedules-19,e99f28b9506e39c1,3d45b0a8c3241dcc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:01:30.884 ERROR 21094 --- [ocp-schedules-21,11427c0d51ec81dc,3cd20f11587224f0] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:80) ... 15 common frames omitted 2025-04-08 21:01:30.884 INFO 21094 --- [ocp-schedules-19,e99f28b9506e39c1,3d45b0a8c3241dcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:01:30.889 INFO 21094 --- [ocp-schedules-18,bb3286a5e33f8154,c777be773de90c18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:01:30.889 INFO 21094 --- [ocp-schedules-7,df3bd79176a92669,6574352fb60f3d65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:01:30.889 ERROR 21094 --- [ocp-schedules-18,bb3286a5e33f8154,c777be773de90c18] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2025-04-08 21:01:30.890 INFO 21094 --- [ocp-schedules-18,bb3286a5e33f8154,c777be773de90c18] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} inited 2025-04-08 21:01:30.891 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:01:30.891 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:01:30.891 INFO 21094 --- [ocp-schedules-2,f4518e0c89ce1adf,879f56e3d8392107] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:01:30.881+08:00, clusters=[2] 2025-04-08 21:01:30.891 INFO 21094 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:01:30.891 INFO 21094 --- [ocp-schedules-4,6c2d8e1fcf80610b,c246d1f6632d3498] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:01:30.891 INFO 21094 --- [ocp-schedules-3,ffbd75cc39c46b9c,6983873703ec372b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:01:30.894 INFO 21094 --- [ocp-schedules-27,6dde0d584bba29f0,c23731a7092e3db4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:01:30.894 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2025-04-08 21:01:30.895 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2025-04-08 21:01:30.895 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:01:30.895 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:01:30.895 INFO 21094 --- [ocp-schedules-25,505fe190e0985cf7,9768135068b9050c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:01:30.896 INFO 21094 --- [ocp-schedules-6,6a708da7da859ef2,b3eda139375fd661] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:01:30.896 INFO 21094 --- [ocp-schedules-11,265e90322de85389,f3564898d9ee1293] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:01:30.896 INFO 21094 --- [ocp-schedules-24,81e11126b8bb1377,519826e5e3cb0304] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:01:30.896 INFO 21094 --- [ocp-schedules-15,7f1613816c9c9439,64866653abcbf22b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:01:30.900 INFO 21094 --- [ocp-schedules-24,81e11126b8bb1377,519826e5e3cb0304] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:01:30.900 INFO 21094 --- [ocp-schedules-11,265e90322de85389,f3564898d9ee1293] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:01:30.900 INFO 21094 --- [ocp-schedules-5,cfc7aafc737d0b98,92c094834b8f5e80] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:01:30.900 INFO 21094 --- [ocp-schedules-6,6a708da7da859ef2,b3eda139375fd661] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:01:30.900 INFO 21094 --- [ocp-schedules-30,854e356e06aad5e6,b337f4d233d1c3fa] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 21:01:30.900 WARN 21094 --- [ocp-schedules-5,cfc7aafc737d0b98,92c094834b8f5e80] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=checkClusterCgroupValue, confName=checkClusterCgroupValue 2025-04-08 21:01:30.900 INFO 21094 --- [ocp-schedules-10,4396cbfceb7df18d,388667ccb4f1bb5e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:01:30.900 ERROR 21094 --- [ocp-schedules-5,cfc7aafc737d0b98,92c094834b8f5e80] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.checkClusterCgroupValue(ClusterSchedules.java:135) ... 15 common frames omitted 2025-04-08 21:01:30.901 WARN 21094 --- [ocp-schedules-25,505fe190e0985cf7,9768135068b9050c] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncAllTenantInfo, confName=syncAllTenantInfo 2025-04-08 21:01:30.901 INFO 21094 --- [ocp-schedules-17,ff7721a8f28cde5b,5f418dec2c90d2e4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:01:30.901 ERROR 21094 --- [ocp-schedules-25,505fe190e0985cf7,9768135068b9050c] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) ... 15 common frames omitted 2025-04-08 21:01:30.901 INFO 21094 --- [ocp-schedules-17,ff7721a8f28cde5b,5f418dec2c90d2e4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:01:30.908 INFO 21094 --- [ocp-schedules-12,a986a008bbb5adf8,44de0da78ecf3e39] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:01:30.909 INFO 21094 --- [ocp-schedules-10,4396cbfceb7df18d,388667ccb4f1bb5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:01:30.909 INFO 21094 --- [ocp-schedules-29,f822de1c57e79b6e,d10061a071c3cbf7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:01:30.910 INFO 21094 --- [ocp-schedules-8,7300df63d092378a,81fc3e7d7fe882f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 21:01:30.910 INFO 21094 --- [ocp-schedules-8,7300df63d092378a,81fc3e7d7fe882f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 21:01:30.910 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:01:30.910 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:01:30.910 INFO 21094 --- [ocp-schedules-14,77d8753ac7e76342,03cd207287bc06cc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:01:30.913 INFO 21094 --- [ocp-schedules-15,7f1613816c9c9439,64866653abcbf22b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:01:30.913 INFO 21094 --- [ocp-schedules-30,854e356e06aad5e6,b337f4d233d1c3fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:01:30.913 INFO 21094 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:01:30.910 INFO 21094 --- [Check status for all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_status_check', labels={app=OB, check_item=cluster connection, failed_reason=-, ob_cluster_id=1744114127, ob_cluster_name=ocpdb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744117290 : 2025-04-08 13:01:30 2025-04-08 21:01:30.916 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2025-04-08 21:01:30.916 INFO 21094 --- [ocp-schedules-2,f4518e0c89ce1adf,879f56e3d8392107] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:01:30.917 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : binlog.cluster.parameter.blacklist= 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/root/ocp-server/log/ocp.log 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/oceanbaselog/sgkyocp/ocp-server.log 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2025-04-08 21:01:30.919 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2025-04-08 21:01:30.922 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:01:30.922 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:01:30.922 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:01:30.923 INFO 21094 --- [ocp-schedules-1,c0cdf11548436b2c,2de7a64efe0bf80f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:01:30.923 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(21094){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 21094 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2025-04-08 21:01:30.923 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2025-04-08 21:01:30.925 INFO 21094 --- [ocp-schedules-28,86454c8a6a3c7688,1a20eca0572dd2c4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:01:30.926 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2025-04-08 21:01:30.928 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2025-04-08 21:01:30.928 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2025-04-08 21:01:30.930 INFO 21094 --- [ocp-schedules-23,ff6a1e2d7ca11ba3,d1707230a359abc5] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 21:01:30.931 INFO 21094 --- [ocp-schedules-21,3cea0a31933ae00d,e988acd083723b94] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:01:30.931 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2025-04-08 21:01:30.931 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2025-04-08 21:01:30.933 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2025-04-08 21:01:30.933 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2025-04-08 21:01:30.933 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2025-04-08 21:01:30.936 INFO 21094 --- [ocp-schedules-5,8a0eb18efe15ea6d,945cb69017dafcd5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2025-04-08 21:01:30.936 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2025-04-08 21:01:30.940 INFO 21094 --- [ocp-schedules-3,e0a2154a61ef496f,37b0e39c507a28cd] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2025-04-08 21:01:30.943 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:01:30.943 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:01:30.943 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:01:30.943 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.force.ssh.installation=false 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80},{"file_regex":"[a-z]+.log.[0-9]+.zst","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.enabled=true 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.run.internal=5m 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.use.specific.http.ip=false 2025-04-08 21:01:30.944 INFO 21094 --- [ocp-async-4,961742af13d2f8ee,5d2fe846ccb57920] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full=4.3.5-20250319105844 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2025-04-08 21:01:30.944 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2025-04-08 21:01:30.945 INFO 21094 --- [ocp-schedules-13,95edde50fd96f963,9bedac78e81e6fbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2025-04-08 21:01:30.946 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2025-04-08 21:01:30.947 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.application.uuid=493eecb387654b6f90f5379d8459859c 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2025-04-08 21:01:30.946 INFO 21094 --- [ocp-async-5,1907eab92e4c6099,80be1d5876bf92bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.object-storage-sdk.invoke.https=true 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2025-04-08 21:01:30.948 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2025-04-08 21:01:30.949 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2025-04-08 21:01:30.949 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2025-04-08 21:01:30.949 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2025-04-08 21:01:30.953 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2025-04-08 21:01:30.953 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2025-04-08 21:01:30.953 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2025-04-08 21:01:30.953 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2025-04-08 21:01:30.953 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2025-04-08 21:01:30.954 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2025-04-08 21:01:30.955 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2025-04-08 21:01:30.955 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2025-04-08 21:01:30.955 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2025-04-08 21:01:30.955 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2025-04-08 21:01:30.955 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/root/ocp/ocp-server/lib 2025-04-08 21:01:30.955 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/root/software 2025-04-08 21:01:30.957 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2025-04-08 21:01:30.957 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2025-04-08 21:01:30.957 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2025-04-08 21:01:30.957 INFO 21094 --- [ocp-schedules-23,ff6a1e2d7ca11ba3,d1707230a359abc5] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 21:01:30.957 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2025-04-08 21:01:30.957 INFO 21094 --- [ocp-schedules-23,ff6a1e2d7ca11ba3,d1707230a359abc5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 21:01:30.957 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2025-04-08 21:01:30.957 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2025-04-08 21:01:30.957 INFO 21094 --- [ocp-schedules-23,ff6a1e2d7ca11ba3,d1707230a359abc5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 21:01:30.958 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2025-04-08 21:01:30.958 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2025-04-08 21:01:30.958 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2025-04-08 21:01:30.960 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2025-04-08 21:01:30.960 INFO 21094 --- [ocp-schedules-28,86454c8a6a3c7688,1a20eca0572dd2c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:01:30.961 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2025-04-08 21:01:30.961 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2025-04-08 21:01:30.961 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2025-04-08 21:01:30.963 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2025-04-08 21:01:30.965 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2025-04-08 21:01:30.965 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/root/ocp/logs/ocp 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name=ocpdb 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2025-04-08 21:01:30.966 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2025-04-08 21:01:30.966 INFO 21094 --- [ocp-schedules-17,ff7721a8f28cde5b,5f418dec2c90d2e4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:01:30.967 INFO 21094 --- [ocp-schedules-17,ff7721a8f28cde5b,5f418dec2c90d2e4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":3000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time=1744114820 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2025-04-08 21:01:30.967 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.rollup.type=MAX 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.service.type=LOCAL 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2025-04-08 21:01:30.968 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=192.168.1.61 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2881 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.batch.replace.arbitration.thread.core-pool-size=100 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist=ocpdb 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-zA-Z_0-9-]{2,64}$ 2025-04-08 21:01:30.969 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2025-04-08 21:01:30.970 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2025-04-08 21:01:30.970 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2025-04-08 21:01:30.970 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2025-04-08 21:01:30.970 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2025-04-08 21:01:30.970 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2025-04-08 21:01:30.970 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obshell.port=2886 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2025-04-08 21:01:30.971 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2025-04-08 21:01:30.972 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2025-04-08 21:01:30.973 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2025-04-08 21:01:30.974 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2025-04-08 21:01:30.975 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://192.168.1.61 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.fixedDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.initialDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2025-04-08 21:01:30.976 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2025-04-08 21:01:30.977 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationAutoReplaceStatusTask.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationServiceGroup.fixedDelay=10 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterCgroupValue.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkSuspendTenantCompactionStatus.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.oracle=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.5 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.5-20250319105844 2025-04-08 21:01:30.978 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8080 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2025-04-08 21:01:30.979 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2025-04-08 21:01:30.980 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2025-04-08 21:01:30.980 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2025-04-08 21:01:30.980 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2025-04-08 21:01:30.987 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2025-04-08 21:01:30.987 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2025-04-08 21:01:30.987 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2025-04-08 21:01:30.987 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2025-04-08 21:01:30.988 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2025-04-08 21:01:30.989 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2025-04-08 21:01:30.989 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2025-04-08 21:01:30.989 INFO 21094 --- [ocp-schedules-12,a986a008bbb5adf8,44de0da78ecf3e39] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2025-04-08 21:01:30.990 INFO 21094 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2025-04-08 21:01:30.991 WARN 21094 --- [ocp-async-7,c62676ba164be29c,cdcb714d40b7dc69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:01:30.997 INFO 21094 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 104.224 seconds (JVM running for 105.112) 2025-04-08 21:01:31.009 INFO 21094 --- [ocp-schedules-3,e0a2154a61ef496f,37b0e39c507a28cd] c.o.o.m.service.SeriesIdCleanService : HostIpSize=1, clusterSize=1, obproxyClusterSize=0 2025-04-08 21:01:31.011 INFO 21094 --- [ocp-schedules-18,bb3286a5e33f8154,c777be773de90c18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:01:31.011 INFO 21094 --- [ocp-schedules-18,bb3286a5e33f8154,c777be773de90c18] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:01:31.011 INFO 21094 --- [ocp-schedules-21,3cea0a31933ae00d,e988acd083723b94] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:01:31.012 INFO 21094 --- [pool-21-thread-1,6a708da7da859ef2,428bdd59dbc0442b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:01:31.027 INFO 21094 --- [pool-overview-container-4,7982469aa4c4d03a,23142a0362b3da7e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:01:31.027 INFO 21094 --- [pool-overview-container-2,b13a9c71be825d28,af5e3a6684856789] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:01:31.041 INFO 21094 --- [ocp-schedules-13,95edde50fd96f963,9bedac78e81e6fbb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:01:31.050 INFO 21094 --- [ocp-schedules-3,e0a2154a61ef496f,37b0e39c507a28cd] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2025-04-08 21:01:31.050 INFO 21094 --- [ocp-schedules-3,e0a2154a61ef496f,37b0e39c507a28cd] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=1 2025-04-08 21:01:31.052 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:01:31.060 INFO 21094 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 21:01:31.060 ERROR 21094 --- [ocp-schedules-10,4396cbfceb7df18d,388667ccb4f1bb5e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$6d8500a1.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:01:31.060 ERROR 21094 --- [ocp-schedules-30,854e356e06aad5e6,b337f4d233d1c3fa] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$6d8500a1.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:01:31.063 INFO 21094 --- [ocp-schedules-10,4396cbfceb7df18d,388667ccb4f1bb5e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:01:31.066 INFO 21094 --- [ocp-schedules-30,854e356e06aad5e6,b337f4d233d1c3fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 21:01:31.068 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:01:31.068 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:01:31.068 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:01:31.068 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:01:31.068 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:01:31.068 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2025-04-08 21:01:31.069 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2025-04-08 21:01:31.075 INFO 21094 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2025-04-08 21:01:31.075 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:01:31.077 INFO 21094 --- [ocp-schedules-3,e0a2154a61ef496f,37b0e39c507a28cd] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=126 2025-04-08 21:01:31.077 INFO 21094 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 493eecb387654b6f90f5379d8459859c. 2025-04-08 21:01:31.077 INFO 21094 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2025-04-08 21:01:31.078 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1M44.309172481S 2025-04-08 21:01:31.078 INFO 21094 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2025-04-08 21:01:31.082 INFO 21094 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2025-04-08 21:01:31.082 INFO 21094 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2025-04-08 21:01:31.083 INFO 21094 --- [ocp-async-7,6b48d57f43643b2b,3410f21cd68c9a9e] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [2] 2025-04-08 21:01:31.084 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=423, maxSyncedRecordId=766200877631795200 2025-04-08 21:01:31.084 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:01:31.084 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:01:31.088 WARN 21094 --- [ocp-schedules-6,6a708da7da859ef2,b3eda139375fd661] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$6d8500a1.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$cb7777f5.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:01:31.088 INFO 21094 --- [ocp-schedules-6,6a708da7da859ef2,b3eda139375fd661] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:01:31.094 WARN 21094 --- [ocp-async-7,6b48d57f43643b2b,3410f21cd68c9a9e] c.o.o.p.a.i.SessionEventSyncService : Fail to refresh cluster event info, cluster id = 2 2025-04-08 21:01:31.094 INFO 21094 --- [ocp-async-7,6b48d57f43643b2b,3410f21cd68c9a9e] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [2] 2025-04-08 21:01:31.094 WARN 21094 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2025-04-08 21:01:31.094 INFO 21094 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2025-04-08 21:01:31.094 INFO 21094 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2025-04-08 21:01:31.109 INFO 21094 --- [ocp-schedules-9,2f9f7021fef6139a,fd5261560a13237c] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=266, maxSyncedRecordId=766198483195916288 2025-04-08 21:01:31.110 ERROR 21094 --- [ocp-async-8,898cfc77bf65fde6,0ce863bc8f9f4774] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2025-04-08 21:01:31.110 INFO 21094 --- [ocp-async-8,898cfc77bf65fde6,0ce863bc8f9f4774] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2025-04-08 21:01:31.121 INFO 21094 --- [pool-11-thread-1,7982469aa4c4d03a,d37d510b9b0724be] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:01:31.121 INFO 21094 --- [pool-11-thread-1,7982469aa4c4d03a,d37d510b9b0724be] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:01:31.121 INFO 21094 --- [pool-11-thread-2,7982469aa4c4d03a,f07adce924126232] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:01:31.123 INFO 21094 --- [pool-11-thread-3,7982469aa4c4d03a,6002dfc73f4ee977] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:01:31.111 INFO 21094 --- [ocp-async-8,898cfc77bf65fde6,0ce863bc8f9f4774] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2025-04-08 21:01:31.129 INFO 21094 --- [pool-11-thread-4,7982469aa4c4d03a,f0b9f67f97906c27] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:01:31.135 INFO 21094 --- [ocp-async-8,898cfc77bf65fde6,0ce863bc8f9f4774] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2025-04-08 21:01:31.136 INFO 21094 --- [pool-11-thread-1,7982469aa4c4d03a,d37d510b9b0724be] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:01:31.137 INFO 21094 --- [ForkJoinPool-7-worker-2,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:01:31.137 INFO 21094 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:01:31.138 INFO 21094 --- [pool-11-thread-1,7982469aa4c4d03a,d37d510b9b0724be] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:01:31.139 INFO 21094 --- [pool-11-thread-2,7982469aa4c4d03a,f07adce924126232] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:01:31.139 INFO 21094 --- [ForkJoinPool-7-worker-9,7982469aa4c4d03a,57767c43638951d1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 21:01:31.139 INFO 21094 --- [ForkJoinPool-7-worker-9,7982469aa4c4d03a,57767c43638951d1] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:01:31.139 INFO 21094 --- [ForkJoinPool-7-worker-9,7982469aa4c4d03a,57767c43638951d1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 21:01:31.139 INFO 21094 --- [ForkJoinPool-7-worker-2,7982469aa4c4d03a,e5b738b884fbab71] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:01:31.139 INFO 21094 --- [pool-11-thread-4,7982469aa4c4d03a,f0b9f67f97906c27] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:01:31.140 INFO 21094 --- [pool-11-thread-3,7982469aa4c4d03a,6002dfc73f4ee977] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:01:31.178 INFO 21094 --- [pool-11-thread-2,7982469aa4c4d03a,f07adce924126232] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:01:31.371 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:01:31.371 INFO 21094 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:01:31.392 INFO 21094 --- [ocp-async-10,629e26ec5d9b3b36,df4b2d4abd1871bf] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2025-04-08 21:01:31.406 INFO 21094 --- [ocp-schedules-4,6c2d8e1fcf80610b,c246d1f6632d3498] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:01:31.411 WARN 21094 --- [ocp-schedules-4,6c2d8e1fcf80610b,c246d1f6632d3498] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:01:31.411 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=ocpdb, obClusterId=1744114127, tenantName=sys, username=root, address=192.168.1.61, port=2881, database=oceanbase) 2025-04-08 21:01:31.411 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:01:31.413 INFO 21094 --- [ocp-schedules-4,6c2d8e1fcf80610b,c246d1f6632d3498] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:01:31.421 WARN 21094 --- [ocp-schedules-4,6c2d8e1fcf80610b,c246d1f6632d3498] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:01:31.424 INFO 21094 --- [obsdk-init-thread-1,ff0f82ff9574dc5b,3422fb199d4fd07a] com.alibaba.druid.pool.DruidDataSource : {dataSource-3,ob-connector-datasource_direct_sys_root_192.168.1.61_2881} inited 2025-04-08 21:01:31.424 INFO 21094 --- [obsdk-init-thread-1,ff0f82ff9574dc5b,3422fb199d4fd07a] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:01:31.424 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:01:31.429 INFO 21094 --- [ocp-schedules-4,6c2d8e1fcf80610b,c246d1f6632d3498] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=1 2025-04-08 21:01:31.435 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:01:31.436 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:01:31.451 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:01:31.506 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:01:31.518 INFO 21094 --- [ocp-async-10,629e26ec5d9b3b36,df4b2d4abd1871bf] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=623 2025-04-08 21:01:31.519 INFO 21094 --- [ocp-async-10,629e26ec5d9b3b36,df4b2d4abd1871bf] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2025-04-08 21:01:31.641 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:01:31.859 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.916258087S 2025-04-08 21:01:31.860 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:01:31.945 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:01:31.946 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:01:31.973 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:01:31.976 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:01:32.074 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:01:32.075 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:01:32.075 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:01:32.075 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:01:32.075 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:01:32.075 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:01:32.083 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:01:32.149 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-1,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2025-04-08 21:01:32.149 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2025-04-08 21:01:32.154 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Completed initialization in 5 ms 2025-04-08 21:01:32.204 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] o.s.boot.SpringApplication : Started application in 1.281 seconds (JVM running for 106.32) 2025-04-08 21:01:32.204 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.282019205S 2025-04-08 21:01:32.220 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:01:32.273 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:01:32.344 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-1,a7a2eb3d81b2893a,6e5e85a5e3186afd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=a7a2eb3d81b2893a, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.347 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-1,a7a2eb3d81b2893a,6e5e85a5e3186afd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=a7a2eb3d81b2893a, duration=157 ms] 2025-04-08 21:01:32.419 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,0ac4325d74ffa781,50530748f8bc4746] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=0ac4325d74ffa781, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.419 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,0ac4325d74ffa781,50530748f8bc4746] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=0ac4325d74ffa781, duration=4 ms] 2025-04-08 21:01:32.428 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,98f0d8bea98b9764,92fd3da233b201b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=98f0d8bea98b9764, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.428 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,98f0d8bea98b9764,92fd3da233b201b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=98f0d8bea98b9764, duration=2 ms] 2025-04-08 21:01:32.434 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,ad4adda79ae203f1,b414ee921542f14b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=ad4adda79ae203f1, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.435 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,ad4adda79ae203f1,b414ee921542f14b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=ad4adda79ae203f1, duration=3 ms] 2025-04-08 21:01:32.442 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,7e8e24110ae00ee9,1c83fe02149d0327] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=7e8e24110ae00ee9, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.442 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,7e8e24110ae00ee9,1c83fe02149d0327] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=7e8e24110ae00ee9, duration=3 ms] 2025-04-08 21:01:32.449 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,69b13f6ec336c771,894768b40422c78d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=69b13f6ec336c771, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.449 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,69b13f6ec336c771,894768b40422c78d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=69b13f6ec336c771, duration=2 ms] 2025-04-08 21:01:32.458 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,dea3f5ec1962ea02,264ce5a35428ec98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=dea3f5ec1962ea02, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.458 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,dea3f5ec1962ea02,264ce5a35428ec98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=dea3f5ec1962ea02, duration=3 ms] 2025-04-08 21:01:32.464 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,85a30f0f1c0360b5,ac624791dae83eb5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=85a30f0f1c0360b5, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.464 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,85a30f0f1c0360b5,ac624791dae83eb5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=85a30f0f1c0360b5, duration=1 ms] 2025-04-08 21:01:32.471 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,11cfd459ac7348a6,135b88f2f9063ec1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=11cfd459ac7348a6, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.471 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,11cfd459ac7348a6,135b88f2f9063ec1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=11cfd459ac7348a6, duration=2 ms] 2025-04-08 21:01:32.477 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,0371ca2a7c169573,6b4672560744f9db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=0371ca2a7c169573, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.477 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,0371ca2a7c169573,6b4672560744f9db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=0371ca2a7c169573, duration=2 ms] 2025-04-08 21:01:32.494 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:01:32.546 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:01:32.569 INFO 21094 --- [ocp-schedules-16,4060681ceff16e9a,95d1ca3c85dbf439] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:01:32.602 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,2970871a0a6eaa26,3ae865abb5ae83df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=2970871a0a6eaa26, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.602 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,2970871a0a6eaa26,3ae865abb5ae83df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=2970871a0a6eaa26, duration=3 ms] 2025-04-08 21:01:32.608 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,0698ea19fe56a3d4,2c1de47ebc0796e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=0698ea19fe56a3d4, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.608 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,0698ea19fe56a3d4,2c1de47ebc0796e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=0698ea19fe56a3d4, duration=1 ms] 2025-04-08 21:01:32.613 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,176ebfe4df58d04e,c3d7a7a1493e3c85] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=176ebfe4df58d04e, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.614 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,176ebfe4df58d04e,c3d7a7a1493e3c85] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=176ebfe4df58d04e, duration=2 ms] 2025-04-08 21:01:32.622 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,e8a0c4a09b2358ed,35077ba2ba37d176] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=e8a0c4a09b2358ed, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.622 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,e8a0c4a09b2358ed,35077ba2ba37d176] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=e8a0c4a09b2358ed, duration=2 ms] 2025-04-08 21:01:32.628 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,74197df4062d96d3,95a1178d8d108d66] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=74197df4062d96d3, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.628 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,74197df4062d96d3,95a1178d8d108d66] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=74197df4062d96d3, duration=2 ms] 2025-04-08 21:01:32.637 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,6059a0ff84142fe2,9ecedd3f6f75b08c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=6059a0ff84142fe2, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.637 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,6059a0ff84142fe2,9ecedd3f6f75b08c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=6059a0ff84142fe2, duration=5 ms] 2025-04-08 21:01:32.644 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,fea2a69a59f8d032,b957eb9e29af06b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=fea2a69a59f8d032, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.644 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,fea2a69a59f8d032,b957eb9e29af06b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=fea2a69a59f8d032, duration=2 ms] 2025-04-08 21:01:32.650 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,d0f629f2ba863548,0e17f016ab9f3d2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=d0f629f2ba863548, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.650 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,d0f629f2ba863548,0e17f016ab9f3d2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=d0f629f2ba863548, duration=2 ms] 2025-04-08 21:01:32.655 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,b5513f23786f0abc,98e7cd6e2623b761] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=b5513f23786f0abc, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.655 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,b5513f23786f0abc,98e7cd6e2623b761] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=b5513f23786f0abc, duration=1 ms] 2025-04-08 21:01:32.660 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,233827d375f1ce2d,d5bfe2b2c19702de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=233827d375f1ce2d, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.660 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,233827d375f1ce2d,d5bfe2b2c19702de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=233827d375f1ce2d, duration=1 ms] 2025-04-08 21:01:32.735 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,ca2de79405185d83,fb5c55cfa6b0a277] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=ca2de79405185d83, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:32.735 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,ca2de79405185d83,fb5c55cfa6b0a277] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=ca2de79405185d83, duration=2 ms] 2025-04-08 21:01:32.759 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:01:32.805 INFO 21094 --- [ocp-async-3,ff0f82ff9574dc5b,ab635a93c1f58e44] o.o.s.g.s.StatsGatherOverviewDetailCache : Sync getNoUpdatedStatsGatherStatisticsMap ok, statisticMap = {} 2025-04-08 21:01:32.896 INFO 21094 --- [ocp-async-5,5a4be4c6f9e47d1d,f99f8da78c87d6b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:01:32.896 WARN 21094 --- [ocp-async-4,bbe463c445cbbaa5,80234501d03b7c48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:01:33.108 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,2d933976ebe438f8,beb9bc8b5e6355b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=2d933976ebe438f8, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.108 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,2d933976ebe438f8,beb9bc8b5e6355b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=2d933976ebe438f8, duration=2 ms] 2025-04-08 21:01:33.113 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,944fc76c82e0da06,de18494a802a10aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=944fc76c82e0da06, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.113 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,944fc76c82e0da06,de18494a802a10aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=944fc76c82e0da06, duration=2 ms] 2025-04-08 21:01:33.118 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,1910cbb13ba4e5eb,aca2989aba76c176] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=1910cbb13ba4e5eb, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.118 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,1910cbb13ba4e5eb,aca2989aba76c176] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=1910cbb13ba4e5eb, duration=1 ms] 2025-04-08 21:01:33.125 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,4179f8a2c1632a44,2ac1cbffdf5ce7e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=4179f8a2c1632a44, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.125 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,4179f8a2c1632a44,2ac1cbffdf5ce7e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=4179f8a2c1632a44, duration=1 ms] 2025-04-08 21:01:33.130 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,8ca33feff6eb30f8,02b6aa45bd613ee2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=8ca33feff6eb30f8, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.130 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,8ca33feff6eb30f8,02b6aa45bd613ee2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=8ca33feff6eb30f8, duration=1 ms] 2025-04-08 21:01:33.136 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,de6f6e012dcfed0c,d38dd18284d84711] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=de6f6e012dcfed0c, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.136 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-2,de6f6e012dcfed0c,d38dd18284d84711] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=de6f6e012dcfed0c, duration=1 ms] 2025-04-08 21:01:33.141 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,16777eb801bf94d9,c2691fffc4fbf499] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=16777eb801bf94d9, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.141 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-5,16777eb801bf94d9,c2691fffc4fbf499] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=16777eb801bf94d9, duration=2 ms] 2025-04-08 21:01:33.146 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,d9fbf33b91ba8f1e,c854f2e563dc2fc9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=d9fbf33b91ba8f1e, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.146 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-7,d9fbf33b91ba8f1e,c854f2e563dc2fc9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=d9fbf33b91ba8f1e, duration=2 ms] 2025-04-08 21:01:33.151 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,f101702f69f2e912,53a6ea6913e255be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=f101702f69f2e912, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.151 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-9,f101702f69f2e912,53a6ea6913e255be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=f101702f69f2e912, duration=2 ms] 2025-04-08 21:01:33.156 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,6098d522a4981f4c,354c927c581145a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=6098d522a4981f4c, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:01:33.156 INFO 21094 --- [http-nio-0.0.0.0-8080-exec-3,6098d522a4981f4c,354c927c581145a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=6098d522a4981f4c, duration=2 ms] 2025-04-08 21:01:46.930 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:01:46.930 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:01:46.930 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:01:46.930 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:01:46.930 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:01:46.931 INFO 26087 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : The following 1 profile is active: "composite" 2025-04-08 21:01:46.939 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:01:52.487 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 21:01:52.488 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 21:01:54.000 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1491 ms. Found 173 JPA repository interfaces. 2025-04-08 21:01:54.006 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 21:01:54.007 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 21:01:54.180 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 172 ms. Found 16 JPA repository interfaces. 2025-04-08 21:01:54.180 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode 2025-04-08 21:01:54.180 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2025-04-08 21:01:54.226 INFO 26087 --- [main,,] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 45 ms. Found 5 JPA repository interfaces. 2025-04-08 21:01:57.110 INFO 26087 --- [main,,] o.s.cloud.context.scope.GenericScope : BeanFactory id=535a2406-c3b5-3b8b-8149-b06173f75dfb 2025-04-08 21:01:57.894 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b682e9' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.149 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2b682e9' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.190 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#668a32a4' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.205 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6182ffef' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.215 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6182ffef' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.242 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration' of type [org.springframework.boot.autoconfigure.sql.init.DataSourceInitializationConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.257 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metadbConfiguration' of type [com.oceanbase.ocp.server.common.config.MetadbConfiguration$$EnhancerBySpringCGLIB$$a852fe2b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.289 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration' of type [org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.297 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.304 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'loadBalancerClientsDefaultsMappingsProvider' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration$$Lambda$657/267400033] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.345 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultsBindHandlerAdvisor' of type [org.springframework.cloud.commons.config.DefaultsBindHandlerAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.365 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpClusterProperty.MetaDbProperties' of type [com.oceanbase.ocp.core.config.OcpClusterProperty$MetaDbProperties$$EnhancerBySpringCGLIB$$40974c3a] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.388 INFO 26087 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 21:01:58.388 INFO 26087 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] username=root@ocp_meta 2025-04-08 21:01:58.388 INFO 26087 --- [main,,] c.o.o.s.c.c.w.MetadbHikariConfigHelper : [metadb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 21:01:58.406 INFO 26087 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Starting... 2025-04-08 21:01:58.456 INFO 26087 --- [main,,] com.zaxxer.hikari.HikariDataSource : metadb-connect-pool - Start completed. 2025-04-08 21:01:58.464 INFO 26087 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 21:01:58.464 INFO 26087 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 21:01:58.465 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaDataSource' of type [com.zaxxer.hikari.HikariDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.492 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.sql.init-org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties' of type [org.springframework.boot.autoconfigure.sql.init.SqlInitializationProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.519 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceScriptDatabaseInitializer' of type [org.springframework.boot.autoconfigure.sql.init.SqlDataSourceScriptDatabaseInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.547 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.565 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa.hibernate-org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.584 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.597 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hikariPoolDataSourceMetadataProvider' of type [org.springframework.boot.autoconfigure.jdbc.metadata.DataSourcePoolMetadataProvidersConfiguration$HikariPoolDataSourceMetadataProviderConfiguration$$Lambda$425/1240412896] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.658 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration' of type [org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.705 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaVendorAdapter' of type [org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.727 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'entityManagerFactoryBuilder' of type [org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:01:58.908 INFO 26087 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 21:01:59.235 INFO 26087 --- [main,,] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.15.Final 2025-04-08 21:02:00.035 INFO 26087 --- [main,,] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} 2025-04-08 21:02:03.002 INFO 26087 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 21:02:04.250 WARN 26087 --- [main,,] o.h.t.d.java.JavaTypeDescriptorRegistry : HHH000481: Encountered Java type [class com.fasterxml.jackson.databind.JsonNode] for which we could not locate a JavaTypeDescriptor and which does not appear to implement equals and/or hashCode. This can lead to significant performance problems when performing equality/dirty checking involving this Java type. Consider registering a custom JavaTypeDescriptor or at least implementing equals/hashCode. 2025-04-08 21:02:05.099 WARN 26087 --- [main,,] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 21:02:05.099 WARN 26087 --- [main,,] org.hibernate.mapping.RootClass : HHH000039: Composite-id class does not override hashCode(): com.oceanbase.ocp.core.metadb.repo.iam.jdbcsession.SpringJdbcSessionAttributeKey 2025-04-08 21:02:08.539 INFO 26087 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 21:02:08.609 INFO 26087 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 21:02:08.625 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:08.641 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaEntityManagerFactory' of type [com.sun.proxy.$Proxy294] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:08.763 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7c4d1c7b' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:08.980 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'propertyManager.MonitorDbExtraProperties' of type [com.oceanbase.ocp.core.property.PropertyManager$MonitorDbExtraProperties$$EnhancerBySpringCGLIB$$fd4dcfcd] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:08.985 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'monitordbConfiguration' of type [com.oceanbase.ocp.server.common.config.MonitordbConfiguration$$EnhancerBySpringCGLIB$$e62202c2] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.053 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'ocpAzoneModelConverter' of type [com.oceanbase.ocp.ha.maz.OcpAzoneModelConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.071 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#765569a4' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.072 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#765569a4' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.073 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#59e0cb10' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.074 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2a40d302' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.074 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2a40d302' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.076 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4ab56032' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.126 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.repository.config.JpaMetamodelMappingContextFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:09.135 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'jpaMappingContext' of type [org.springframework.data.jpa.mapping.JpaMetamodelMappingContext] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.085 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.101 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationResourceRepository' of type [com.sun.proxy.$Proxy300] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.155 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3db55907' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.156 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3db55907' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.157 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#65ccae11' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.158 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#72bdb6d3' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.158 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#72bdb6d3' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.159 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#74131bff' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.528 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.529 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOverviewRepository' of type [com.sun.proxy.$Proxy304] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.557 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#73b6167d' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.558 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#73b6167d' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.559 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6746c3ef' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.560 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43b54374' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.560 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#43b54374' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.561 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#11fe8a70' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.588 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.589 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleOverviewRepository' of type [com.sun.proxy.$Proxy305] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.609 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#63fac644' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.609 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#63fac644' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.610 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1fdedffb' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.611 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#71190244' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.611 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#71190244' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.612 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3255707f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.648 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.649 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'organizationRepository' of type [com.sun.proxy.$Proxy306] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.682 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2898a250' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.683 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2898a250' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.684 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1088a4f0' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.684 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#33c9eb17' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.685 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#33c9eb17' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.686 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f69187c' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.710 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.711 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionGroupRepository' of type [com.sun.proxy.$Proxy307] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:10.786 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'metaTransactionManager' of type [org.springframework.orm.jpa.JpaTransactionManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.065 INFO 26087 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action group type info: {"HOST_MANAGE":"HOST","ALL":"UNKNOWN","HOST_MAINTAIN":"HOST","ALL_READ":"UNKNOWN","OBPROXY_MAINTAIN":"OBPROXY","ORGANIZATION_READ":"ORGANIZATION","HOST_CREATE":"HOST","USER_MANAGE":"USER","ARBITRATION_READ":"ARBITRATION","ARBITRATION_CREATE":"ARBITRATION","BINLOG_MANAGE":"BINLOG","OBPROXY_MANAGE":"OBPROXY","HOST_READ":"HOST","BINLOG_READ":"BINLOG","AUDIT_READ":"AUDIT","USER_READ":"USER","ROLE_READ":"ROLE","CLUSTER_MANAGE":"CLUSTER","CLUSTER_CREATE":"CLUSTER","TENANT_MANAGE":"TENANT","CLUSTER_MAINTAIN":"CLUSTER","TENANT_MAINTAIN":"TENANT","CLUSTER_READ":"CLUSTER","BINLOG_MAINTAIN":"BINLOG","TENANT_READ":"TENANT","OBPROXY_READ":"OBPROXY","ARBITRATION_MAINTAIN":"ARBITRATION","ROLE_MANAGE":"ROLE","BINLOG_CREATE":"BINLOG","ARBITRATION_MANAGE":"ARBITRATION","OBPROXY_CREATE":"OBPROXY"} 2025-04-08 21:02:11.067 INFO 26087 --- [main,,] c.o.o.c.s.a.r.helper.ActionMetaHelper : Action type info: {"ALARM:CREATE":"ALARM","SELFCURE:UPDATE":"SELFCURE","HOST:DELETE":"HOST","ROLE:CREATE":"ROLE","HOST:UPDATE":"HOST","CLUSTER:UPDATE":"CLUSTER","INSPECTION:UPDATE":"INSPECTION","CLUSTER:READ":"CLUSTER","PACKAGE:READ":"PACKAGE","ALARM:READ":"ALARM","TAG:DELETE":"TAG","BINLOG:CREATE":"BINLOG","ARBITRATION:CREATE":"ARBITRATION","SELFCURE:READ":"SELFCURE","*":"UNKNOWN","CLUSTER:DELETE":"CLUSTER","CREDENTIAL:DELETE":"CREDENTIAL","OBPROXY:READ":"OBPROXY","PROPERTY:READ":"PROPERTY","ARBITRATION:UPDATE":"ARBITRATION","OBPROXY:DELETE":"OBPROXY","TENANT:UPDATE":"TENANT","BINLOG:READ":"BINLOG","HOST:READ":"HOST","INSPECTION:CREATE":"INSPECTION","ALARM:UPDATE":"ALARM","PROPERTY:UPDATE":"PROPERTY","BINLOG:UPDATE":"BINLOG","PACKAGE:UPDATE":"PACKAGE","HOST:CREATE":"HOST","ROLE:DELETE":"ROLE","TAG:READ":"TAG","ROLE:READ":"ROLE","BINLOG:DELETE":"BINLOG","TASK:UPDATE":"TASK","USER:DELETE":"USER","CREDENTIAL:UPDATE":"CREDENTIAL","USER:UPDATE":"USER","TENANT:READ":"TENANT","TAG:UPDATE":"TAG","OBPROXY:CREATE":"OBPROXY","ARBITRATION:READ":"ARBITRATION","ROLE:UPDATE":"ROLE","ORGANIZATION:READ":"ORGANIZATION","TENANT:DELETE":"TENANT","USER:READ":"USER","AUDIT:READ":"AUDIT","TASK:READ":"TASK","TAG:CREATE":"TAG","INSPECTION:READ":"INSPECTION","ARBITRATION:DELETE":"ARBITRATION","USER:CREATE":"USER","PACKAGE:DELETE":"PACKAGE","ALARM:DELETE":"ALARM","CREDENTIAL:READ":"CREDENTIAL","OBPROXY:UPDATE":"OBPROXY","CREDENTIAL:CREATE":"CREDENTIAL","CLUSTER:CREATE":"CLUSTER"} 2025-04-08 21:02:11.067 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'actionMetaHelper' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ActionMetaHelper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.093 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'defaultPerfConfiguration' of type [com.oceanbase.ocp.server.common.config.DefaultPerfConfiguration$$EnhancerBySpringCGLIB$$857650d0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.128 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'authenticationFacadeImpl' of type [com.oceanbase.ocp.core.security.AuthenticationFacadeImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.154 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ee60495' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.154 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6ee60495' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.155 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#466eddc6' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.155 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75321ba4' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.155 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75321ba4' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.156 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#471a163a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.244 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.245 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOverviewRepository' of type [com.sun.proxy.$Proxy310] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.265 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostOperator' of type [com.oceanbase.ocp.core.security.organization.operator.HostOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.283 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f02369f' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.283 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#f02369f' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.284 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2d03be3' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.284 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#76d6769a' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.285 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#76d6769a' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.287 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6c6f2c27' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.349 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.350 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOverviewRepository' of type [com.sun.proxy.$Proxy312] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.366 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obClusterOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObClusterOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.383 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1f259685' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.383 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1f259685' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.384 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3d6aa682' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.384 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#21378791' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.384 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#21378791' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.385 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4a241c5d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.445 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.446 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyClusterRepository' of type [com.sun.proxy.$Proxy313] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.468 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObproxyOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.486 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6d0e2a12' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.486 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6d0e2a12' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.487 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4d0f58a9' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.487 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6834f33a' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.487 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6834f33a' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.488 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2858d667' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.528 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.529 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOverviewRepository' of type [com.sun.proxy.$Proxy315] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.542 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ObTenantOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.559 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7c2c3098' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.559 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7c2c3098' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.560 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#258b584c' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.561 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2c2a8c3d' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.561 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2c2a8c3d' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.562 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7859c88f' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.619 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.620 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userRepository' of type [com.sun.proxy.$Proxy316] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.643 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'userOperator' of type [com.oceanbase.ocp.core.security.organization.operator.UserOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.659 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#eb5589' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.659 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#eb5589' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.660 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e39cbaa' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.660 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3467986' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.661 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3467986' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.662 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7fdbe204' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.687 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.688 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceOverviewRepository' of type [com.sun.proxy.$Proxy318] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.706 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationOperator' of type [com.oceanbase.ocp.core.security.organization.operator.ArbitrationOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.721 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2419f457' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.722 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2419f457' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.722 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7aee9f49' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.723 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d03099a' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.723 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1d03099a' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.724 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#72112133' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.760 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.761 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogClusterRepository' of type [com.sun.proxy.$Proxy319] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.776 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogOperator' of type [com.oceanbase.ocp.core.security.organization.operator.BinlogOperator] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.778 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceManager' of type [com.oceanbase.ocp.core.security.organization.ResourceManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.792 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#36b085da' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.792 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#36b085da' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.793 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#341df52' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.794 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#15f4dffe' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.794 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#15f4dffe' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.795 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#7f91d389' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.846 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.848 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleRepository' of type [com.sun.proxy.$Proxy320] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.885 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#8c790e7' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.885 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#8c790e7' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.886 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1bad6995' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.887 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4f1f569a' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.887 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4f1f569a' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.888 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1ad30f6e' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.908 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.909 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'rolePermissionRepository' of type [com.sun.proxy.$Proxy322] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.932 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75806ed6' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.932 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#75806ed6' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.934 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#5a4c9b33' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.934 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#375749fe' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.935 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#375749fe' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.937 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#53d458db' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.978 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.979 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'roleMenuRepository' of type [com.sun.proxy.$Proxy323] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:11.991 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'getOrganizationService' of type [com.oceanbase.ocp.core.security.organization.OrganizationServiceImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.019 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#29d015b4' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.019 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#29d015b4' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.020 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#30817cdc' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.020 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#90772f7' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.020 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#90772f7' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.022 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#26b9269d' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.125 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.125 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obZoneRepository' of type [com.sun.proxy.$Proxy324] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.150 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#25e0a88c' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.150 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#25e0a88c' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.150 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6bf666e0' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.151 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c47eb9b' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.151 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4c47eb9b' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.153 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#2bdf341a' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.182 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.182 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obServerOverviewRepository' of type [com.sun.proxy.$Proxy325] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.203 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#60dde2c2' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.203 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#60dde2c2' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.204 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6b52ff96' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.204 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1fafbd29' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.204 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#1fafbd29' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.205 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#a058884' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.265 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.266 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceRepository' of type [com.sun.proxy.$Proxy326] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.289 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e189ee3' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.290 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6e189ee3' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.290 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#61a309fe' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.291 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#456bff44' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.291 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#456bff44' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.292 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6a2158a2' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.322 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.322 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationServiceGroupRepository' of type [com.sun.proxy.$Proxy327] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.338 INFO 26087 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:02:12.459 INFO 26087 --- [main,,] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:02:12.460 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.479 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#63730298' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.479 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#63730298' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.480 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6efa105a' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.480 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3adb51f2' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.480 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#3adb51f2' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.481 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#586e39e1' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.520 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.520 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyServerRepository' of type [com.sun.proxy.$Proxy330] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.538 INFO 26087 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:02:12.549 INFO 26087 --- [main,,] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:02:12.549 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obProxyQueryCache' of type [com.oceanbase.ocp.core.cache.helper.ObProxyQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.566 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6fac8b24' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.567 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#6fac8b24' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.567 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#755fbfab' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.568 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#57a89e68' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.568 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#57a89e68' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.569 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#53fd4420' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.590 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.591 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogServerRepository' of type [com.sun.proxy.$Proxy331] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.611 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#309c39dc' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.612 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#309c39dc' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.613 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#54b5b046' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.613 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4d583069' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.613 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#4d583069' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.614 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean '(inner bean)#29863c20' of type [com.sun.proxy.$Proxy296] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.650 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.651 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obTenantBinlogServiceRepository' of type [com.sun.proxy.$Proxy332] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.666 INFO 26087 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:02:12.682 INFO 26087 --- [main,,] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:02:12.683 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogQueryCache' of type [com.oceanbase.ocp.core.cache.helper.BinlogQueryCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.685 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceCache' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.helper.ResourceCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.687 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'hostResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.697 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'adminResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.706 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'clusterResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.716 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'arbitrationResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.724 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'obproxyResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.733 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'binlogResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.743 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'tenantResourceSupplierImpl' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.745 INFO 26087 --- [main,,] c.o.o.c.s.a.r.ResourceSupplierManager : Resource supplier list: [com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.HostResourceSupplierImpl@22c73577, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.AdminResourceSupplierImpl@5e9fb3b7, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ClusterResourceSupplierImpl@7fce8421, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ArbitrationResourceSupplierImpl@3aa6fef3, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.ObproxyResourceSupplierImpl@2acf4eed, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.BinlogResourceSupplierImpl@654a4b51, com.oceanbase.ocp.core.security.acl.resourcesupplier.impl.TenantResourceSupplierImpl@2a6b4454] 2025-04-08 21:02:12.745 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'resourceSupplierManager' of type [com.oceanbase.ocp.core.security.acl.resourcesupplier.ResourceSupplierManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.773 INFO 26087 --- [main,,] trationDelegate$BeanPostProcessorChecker : Bean 'commonSecurityConfiguration.MethodSecurityConfiguration' of type [com.oceanbase.ocp.server.common.config.security.conf.CommonSecurityConfiguration$MethodSecurityConfiguration$$EnhancerBySpringCGLIB$$b462cd58] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2025-04-08 21:02:12.790 INFO 26087 --- [main,,] .o.s.c.c.s.c.CommonSecurityConfiguration : Method-level access control enabled. 2025-04-08 21:02:13.237 INFO 26087 --- [main,,] c.o.o.c.distributed.server.LocalServer : Local server, id=192.168.1.61:8080, name=b88d8e1c7d 2025-04-08 21:02:13.259 WARN 26087 --- [main,,] c.o.o.c.d.server.DefaultServerUpdater : Deleted expired server: DistributedServer{id=192.168.1.61:8080, name=b88d8e1c7d, expireInSeconds=30, createTime=2025-04-08T21:00:24+08:00, updateTime=2025-04-08T21:01:29+08:00, expireTime=2025-04-08T21:01:59+08:00, isActive=false} 2025-04-08 21:02:13.282 INFO 26087 --- [main,,] c.o.o.c.d.s.DistributedServerManager : DistributedServer updater scheduler started on 192.168.1.61:8080 2025-04-08 21:02:14.107 INFO 26087 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2025-04-08 21:02:14.141 INFO 26087 --- [main,,] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2025-04-08 21:02:14.141 INFO 26087 --- [main,,] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.98] 2025-04-08 21:02:14.308 INFO 26087 --- [main,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2025-04-08 21:02:14.308 INFO 26087 --- [main,,] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 27369 ms 2025-04-08 21:02:14.671 INFO 26087 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : ocp-monitor bean configuration done 2025-04-08 21:02:14.920 INFO 26087 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] driver class=com.oceanbase.jdbc.Driver 2025-04-08 21:02:14.920 INFO 26087 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] username=root@ocp_meta 2025-04-08 21:02:14.920 INFO 26087 --- [main,,] .o.s.c.c.w.ConfigServerDataSourceWrapper : [config server datasource] jdbc url=jdbc:oceanbase://192.168.1.61:2881/meta_database?useUnicode=true&characterEncoding=UTF8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=30000&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull 2025-04-08 21:02:14.952 INFO 26087 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 21:02:14.952 INFO 26087 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 21:02:15.490 INFO 26087 --- [main,,] c.o.ocp.ha.maz.OcpMultiAzoneServiceImpl : OCP multi-azone not enabled 2025-04-08 21:02:15.502 INFO 26087 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : host 192.168.1.61, port 2881, database monitor_database 2025-04-08 21:02:15.503 INFO 26087 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] driver class=com.oceanbase.jdbc.Driver 2025-04-08 21:02:15.503 INFO 26087 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] username=root@ocp_monitor 2025-04-08 21:02:15.503 INFO 26087 --- [main,,] .o.o.s.c.c.w.MonitordbHikariConfigHelper : [monitordb] jdbc url=jdbc:oceanbase://192.168.1.61:2881/monitor_database?useUnicode=true&characterEncoding=utf-8&encloseParamInParentheses=false&connectTimeout=2000&socketTimeout=120000&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=convertToNull 2025-04-08 21:02:15.503 INFO 26087 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Starting... 2025-04-08 21:02:15.507 INFO 26087 --- [main,,] com.zaxxer.hikari.HikariDataSource : monitor-connect-pool - Start completed. 2025-04-08 21:02:15.508 INFO 26087 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : begin AfterDataSourceCreationHook 2025-04-08 21:02:15.508 INFO 26087 --- [main,,] c.o.o.b.spring.DataSourceInterceptor : end AfterDataSourceCreationHook 2025-04-08 21:02:16.123 INFO 26087 --- [main,,] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] 2025-04-08 21:02:18.329 INFO 26087 --- [main,,] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect 2025-04-08 21:02:18.755 INFO 26087 --- [main,,] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] 2025-04-08 21:02:18.756 INFO 26087 --- [main,,] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' 2025-04-08 21:02:31.691 INFO 26087 --- [main,,] org.reflections.Reflections : Reflections took 4318 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 21:02:36.161 INFO 26087 --- [main,,] org.reflections.Reflections : Reflections took 4410 ms to scan 83 urls, producing 1093 keys and 7856 values 2025-04-08 21:02:40.289 INFO 26087 --- [main,,] org.reflections.Reflections : Reflections took 4118 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 21:02:40.302 INFO 26087 --- [main,,] c.o.o.s.c.analyzer.OcpStartUpChecker : audit event is disable 2025-04-08 21:02:40.481 INFO 26087 --- [main,,] c.o.o.alarm.schedule.AlarmScheduledJobs : AlarmSchedules construct done 2025-04-08 21:02:40.927 INFO 26087 --- [main,,] c.o.o.a.c.AlarmConfigurationManager : defaultTimeZone not set, use system default, zoneId=Asia/Shanghai 2025-04-08 21:02:41.503 INFO 26087 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : scheduleEnabled=true 2025-04-08 21:02:41.528 INFO 26087 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize global tasks done 2025-04-08 21:02:41.528 INFO 26087 --- [main,,] c.o.o.a.s.OcpAlarmInitializeService : initialize done 2025-04-08 21:02:42.332 INFO 26087 --- [main,,] c.o.o.alarm.core.silence.GlobalSilencer : enabled alarm types changed, previous=, current=arbitration_group_auto_replace_failed,ob_log_clog_write_error,ob_log_right_to_die,ob_host_net_send_percent_over_threshold,binlog_host_storage_usage_over_threshold,oblb_etcd_leader_absent,ob_tenant_compaction_error,ob_log_easy_hold_timeout,odp_instance_expanse_failed,binlog_host_cpu_usage_over_threshold,ob_log_decent_voting,ob_cluster_frozen_version_delta_over_threshold,binlog_node_status_abnormal,ob_log_commit_error,obproxy_memory_peak,ob_log_mysql_request_full,ob_log_timer_cost_much,ob_log_worker_thread_full,host_disk_readonly,node_file_root_usage,rootservice_log_alarm,ob_log_disk_almost_full,obdns_etcd_cannot_connect_to_leader,oms_sync_failed,binlog_service_conversion_delay,ob_cluster_sync_delay_time_too_long,arbitration_service_unavailable,ob_cluster_merge_timeout,ob_common_unexpected_internal_error,partition_create_failed,ob_cluster_recyclebin_disk_used_over_threshold,ob_log_mem_alloc_error,ob_rootservice_stop_fail,base_backup_timeout,refresh_location_cache_failed,host_ntp_offset_too_large,ob_log_error_lib_direct_alloc,ob_host_invalid_disk_exists,ob_cluster_no_frozen,ocp_collect_metric_failure_rate_high,ob_log_distributed_strong_read,ob_tenant_cpu_usage_over_threshold,ob_log_fail_check_localserver,host_agent_open_fd_count_over_threshold,arbitration_auto_replace_operation_info,ob_cluster_sync_failed,oblb_ip_forward_invalid,backup_process_dead,oblb_keepalived_service_inactive,oblb_process_dead,ob_tenant500_storage_short_meta_mem_hold_high,ob_host_install_disk_percent_over_threshold,ob_tenant500_mem_hold_percent_over_threshold,odp_sql_execute_failed,obagent_dead,ob_server_stop_fail,ob_server_cannot_connect_arbitration,ob_log_rpc_error,ob_server_sstable_percent_over_threshold,oms_sync_delay,ob_host_monitordb_disconnected,oblb_keepalived_become_master,os_tsar_traffic_drop,ob_log_unit_migrate_error,ob_log_reply_memstore_failed,host_agent_version_not_same,ob_log_global_trans_version_delay,ob_tenant_partition_leader_absent,ob_cluster_merge_error,ob_tenant_name_mismatch, 2025-04-08 21:02:43.274 INFO 26087 --- [main,,] c.o.o.monitor.MonitorBeanConfiguration : Table switch time is 1744114820 2025-04-08 21:02:44.459 INFO 26087 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:02:44.511 INFO 26087 --- [main,,] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:02:44.523 INFO 26087 --- [main,,] c.o.o.m.query.LocalMetricQueryService : Monitor router init, username=system, connectTimeoutMs=3000, readTimeoutMs=120000 2025-04-08 21:02:45.497 INFO 26087 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion 2025-04-08 21:02:45.504 INFO 26087 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:02:45.504 INFO 26087 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion 2025-04-08 21:02:45.504 INFO 26087 --- [main,,] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:02:45.505 INFO 26087 --- [main,,] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:02:45.963 INFO 26087 --- [main,,] c.o.o.a.c.n.NotificationSender : NotificationSender init, scriptDirectoryPath=/root/ocp-server/alarm 2025-04-08 21:02:46.151 INFO 26087 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmTaskExecutor 2025-04-08 21:02:46.203 INFO 26087 --- [main,,] c.o.o.alarm.utils.task.AlarmTaskManager : alarm task manager construct done, executorName=192.168.1.61:26087 2025-04-08 21:02:46.277 INFO 26087 --- [main,,] c.o.o.a.c.d.metrics.AlarmMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 21:02:46.388 INFO 26087 --- [main,,] c.o.o.a.c.d.metrics.SqlMetricsQuery : createMetricMetaMapping done, metasCount=172 2025-04-08 21:02:46.408 INFO 26087 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize alarmDetectMetricExecutor 2025-04-08 21:02:46.448 INFO 26087 --- [main,,] c.o.ocp.alarm.core.task.AlarmDetectTask : AlarmDetectTask created 2025-04-08 21:02:46.727 INFO 26087 --- [main,,] c.o.o.s.common.config.BeanConfiguration : begin set auto discover false. 2025-04-08 21:02:46.730 INFO 26087 --- [main,,] c.o.o.s.common.config.BeanConfiguration : set auto discover false success. 2025-04-08 21:02:46.819 INFO 26087 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=2, heartbeatTime=2025-04-08T21:01:31+08:00, now=2025-04-08T21:02:46.818+08:00 2025-04-08 21:02:46.822 INFO 26087 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Consumer offline, id=1, heartbeatTime=2025-04-08T21:01:31+08:00, now=2025-04-08T21:02:46.822+08:00 2025-04-08 21:02:46.950 INFO 26087 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-target, name=192.168.1.61:8080, consumerId=1, resources=[ShardingSlice(consumerId=1, type=OBCluster, resourceIds=[2]), ShardingSlice(consumerId=1, type=Host, resourceIds=[2])] 2025-04-08 21:02:46.950 INFO 26087 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=alarm-detect-by-rule, name=192.168.1.61:8080, consumerId=2, resources=[ShardingSlice(consumerId=2, type=DEFAULT, resourceIds=[18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 80, 81, 106, 107, 250, 251, 252, 253, 254])] 2025-04-08 21:02:47.071 INFO 26087 --- [main,,] com.oceanbase.ocp.core.util.BeanUtils : BeanUtils initialized 2025-04-08 21:02:47.590 INFO 26087 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances start. 2025-04-08 21:02:47.624 INFO 26087 --- [tag-manual-task-1,,] c.o.ocp.service.task.TaskTagService : Tag task instances end. 2025-04-08 21:02:47.833 INFO 26087 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : FileStorageService init, fileDirPath=/home/root/software 2025-04-08 21:02:49.953 INFO 26087 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from OFFLINE to NEW. 2025-04-08 21:02:50.203 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.print.sql = true 2025-04-08 21:02:50.204 INFO 26087 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set sqlLogEnabled=true 2025-04-08 21:02:50.205 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.max-idle.seconds = 3600 2025-04-08 21:02:50.219 INFO 26087 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set maxIdleSeconds=3600 2025-04-08 21:02:50.222 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.cleanup.period.seconds = 300 2025-04-08 21:02:50.222 INFO 26087 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set cleanPeriod=300 2025-04-08 21:02:50.224 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.query.global.timeout.millis = 10000 2025-04-08 21:02:50.224 INFO 26087 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set queryGlobalTimeoutMillis=10000 2025-04-08 21:02:50.225 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.operation.global.timeout.millis = 300000 2025-04-08 21:02:50.225 INFO 26087 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set ddlGlobalTimeoutMillis=300000 2025-04-08 21:02:50.226 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.timeout.millis = 10000 2025-04-08 21:02:50.226 INFO 26087 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set initTimeoutMillis=10000 2025-04-08 21:02:50.228 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.slow.query.threshold.millis = 1000 2025-04-08 21:02:50.229 INFO 26087 --- [main,,] c.o.ocp.obsdk.config.Configurations : [obsdk] set slowQueryThresholdMillis=1000 2025-04-08 21:02:50.230 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.holder.capacity = 2000 2025-04-08 21:02:50.232 INFO 26087 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set cacheSize=2000 2025-04-08 21:02:50.233 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.read.timeout.millis = 1800000 2025-04-08 21:02:50.233 INFO 26087 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketReadTimeoutMillis=1800000 2025-04-08 21:02:50.234 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.connector.init.executor.thread-count = 16 2025-04-08 21:02:50.234 INFO 26087 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketInitThreadCount=16 2025-04-08 21:02:50.235 INFO 26087 --- [main,,] c.o.o.s.ob.obsdk.config.ObSdkConfig : [ObSdkConfig] init, obsdk.socket.connect.timeout.millis = 2000 2025-04-08 21:02:50.235 INFO 26087 --- [main,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] set socketConnectTimeoutMillis=2000 2025-04-08 21:02:51.962 INFO 26087 --- [sharding-alarm-detect-by-target1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=2, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByTargetIdShardingStrategy@7f842c47 2025-04-08 21:02:51.964 INFO 26087 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 21:02:51.974 INFO 26087 --- [sharding-alarm-detect-by-rule1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=1, strategy=com.oceanbase.ocp.alarm.sharding.AlarmDetectByRuleShardingStrategy@541f4873 2025-04-08 21:02:51.975 INFO 26087 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 21:02:52.960 INFO 26087 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 21:02:52.960 INFO 26087 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 21:02:53.072 INFO 26087 --- [sharding-alarm-detect-by-rule1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 21:02:53.073 INFO 26087 --- [sharding-alarm-detect-by-target1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 21:02:55.966 INFO 26087 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 21:02:55.966 INFO 26087 --- [heartbeat-alarm-detect-by-rule-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 21:02:55.971 INFO 26087 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 21:02:55.971 INFO 26087 --- [heartbeat-alarm-detect-by-target-192.168.1.61:80801,,] c.o.o.alarm.AlarmShardingConfiguration : Consumer had rebalanced. 2025-04-08 21:02:56.274 INFO 26087 --- [main,,] org.reflections.Reflections : Reflections took 4049 ms to scan 83 urls, producing 1012 keys and 5046 values 2025-04-08 21:02:56.285 INFO 26087 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116021] DistributedEventManager scanned event classes: [class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent, class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent, class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent, class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent, class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent] 2025-04-08 21:02:56.285 INFO 26087 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.NoContentDistributedEvent failed, error: null 2025-04-08 21:02:56.286 INFO 26087 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.SimpleDistributedEvent failed, error: null 2025-04-08 21:02:56.286 INFO 26087 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116025] DistributedEventManager construct event class class com.oceanbase.ocp.core.distributed.event.KeyValueDistributedEvent failed, error: null 2025-04-08 21:02:56.286 INFO 26087 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116022] DistributedEventManager scanned event class map: {refresh ocp cluster cache=class com.oceanbase.ocp.core.mc.cache.RefreshOcpClusterCacheEvent, reset-all-obproxy-id=class com.oceanbase.ocp.service.obproxy.event.ResetAllProxyIdEvent, refresh-config-server-cache=class com.oceanbase.ocp.core.configurl.event.RefreshClusterCacheEvent, tenant-renamed=class com.oceanbase.ocp.obops.event.TenantRenamedDistributedEvent, invalidate-ob-version-cache=class com.oceanbase.ocp.obops.cluster.event.InvalidateObVersionEvent} 2025-04-08 21:02:56.303 INFO 26087 --- [main,,] c.o.o.c.d.event.DistributedEventManager : [OCP116001] DistributedEventManager started on b88d8e1c7d 2025-04-08 21:02:57.226 INFO 26087 --- [main,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Init eventTemplateCache: 172 cache entries, 172 elements. 2025-04-08 21:02:57.853 INFO 26087 --- [main,,] c.o.o.s.g.prefs.PrefsParameterService : Start loading prefs template. 2025-04-08 21:02:58.261 INFO 26087 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Begin init binlog cluster parameter info. 2025-04-08 21:02:58.271 INFO 26087 --- [main,,] c.o.o.b.i.BinlogClusterParameterService : Finish init binlog cluster parameter info, parameter blacklist: [] 2025-04-08 21:02:58.319 INFO 26087 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Begin init binlog instance parameter info. 2025-04-08 21:02:58.324 INFO 26087 --- [main,,] c.o.o.b.i.BinlogInstanceParameterService : Finish init binlog instance parameter info, parameter blacklist: [] 2025-04-08 21:02:59.242 INFO 26087 --- [main,,] c.o.o.a.c.AlarmConfiguration : alarm configuration set from system properties 2025-04-08 21:03:00.277 INFO 26087 --- [main,,] c.o.o.s.c.analyzer.SslTrustInitializer : ssl not enabled, do nothing 2025-04-08 21:03:00.311 INFO 26087 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Validate and set keys, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB, valid=true 2025-04-08 21:03:00.311 INFO 26087 --- [main,,] c.o.o.s.c.c.s.u.SensitiveDataDecryptor : Sensitive data encryption key is legal. 2025-04-08 21:03:00.422 INFO 26087 --- [main,,] c.o.o.c.t.engine.util.ThreadPrintStream : replaceSystemOut done 2025-04-08 21:03:00.674 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 21:03:00.737 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 21:03:00.758 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:03:00.767 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:03:01.981 INFO 26087 --- [main,,] c.o.o.c.d.s.DistributedServiceManager : DistributedService updater scheduler started on 192.168.1.61:8080 2025-04-08 21:03:01.991 INFO 26087 --- [main,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Init, apiConnectTimeoutMillis:1000, apiReadTimeoutMillis:1000, taskSchedulePeriodSec:3 2025-04-08 21:03:02.063 WARN 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 21:03:02.063 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 21:03:02.419 INFO 26087 --- [main,,] c.o.o.m.promethues.InnerMetricContainer : init inner metric container success, configToMetricMap = {InnerMetricContainer.InnerMetricConfig(scrapeType=second)=[system_cpu_count, jvm_threads_states_threads, jvm_gc_memory_promoted_bytes_total, jvm_memory_committed_bytes, ocp_monitor_collect_request_errors_total, ocp_meta_db_health, jvm_gc_pause_seconds_max, ocp_monitor_collect_request_duration_ms_count, system_cpu_usage, process_threads, http_server_requests_seconds_count, jvm_gc_memory_allocated_bytes_total, jvm_threads_peak_threads, ocp_alarm_detect_duration_ms_count, jvm_memory_used_bytes, process_cpu_usage, jvm_memory_max_bytes, jvm_gc_max_data_size_bytes, http_server_requests_seconds_sum, jvm_gc_pause_seconds_sum, ocp_alarm_detect_duration_ms_sum, jvm_threads_live_threads, system_load_average_1m, jvm_gc_pause_seconds_count, jvm_gc_overhead_percent]} 2025-04-08 21:03:02.432 INFO 26087 --- [main,,] c.o.o.s.c.config.MetricConfiguration : Monitor sharding strategy is monitor 2025-04-08 21:03:04.049 INFO 26087 --- [main,,] c.o.o.s.c.BuiltInFilesInitializeLoader : Load lib files, localBuiltInDir=/root/ocp/ocp-server/lib 2025-04-08 21:03:04.049 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Remove built-in file:obproxyd.sh 2025-04-08 21:03:04.079 INFO 26087 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Delete object, bucket=built-in, name=obproxyd.sh, objectId=1000002 2025-04-08 21:03:04.095 INFO 26087 --- [main,,] c.o.o.s.o.operator.ObjectBlockOperator : deleteByFileId, deletedBlockCount=1 2025-04-08 21:03:04.118 INFO 26087 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh.sha1sum 2025-04-08 21:03:04.118 INFO 26087 --- [main,,] c.o.o.s.o.operator.LocalFileOperator : delete local file, absolutePathName=/home/root/software/built-in/obproxyd.sh 2025-04-08 21:03:04.119 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Load built-in file:obproxyd.sh 2025-04-08 21:03:04.162 INFO 26087 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=200, bucket=built-in, name=obproxyd.sh, objectId=1000003 2025-04-08 21:03:04.211 INFO 26087 --- [main,,] .o.o.s.o.i.LocalObjectStorageServiceImpl : Finish saving object, objectId=1000003, name=obproxyd.sh 2025-04-08 21:03:04.226 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 21:03:04.250 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm 2025-04-08 21:03:04.250 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 21:03:04.253 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=ocp-agent-ce-4.3.5-20250319105844.el7.aarch64.rpm 2025-04-08 21:03:04.253 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:03:04.256 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-libs-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:03:04.256 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:03:04.259 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=oceanbase-ce-utils-4.3.5.1-101000042025031818.el8.x86_64.rpm 2025-04-08 21:03:04.259 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : packageName:obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 21:03:04.261 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Software package already exist, name=obproxy-ce-4.3.3.0-5.el8.x86_64.rpm 2025-04-08 21:03:04.263 INFO 26087 --- [main,,] c.o.o.s.s.b.BuiltInFileServiceImpl : Save matched ocp-agent version, version=4.3.5-20250319105844 2025-04-08 21:03:04.268 INFO 26087 --- [main,,] c.o.ocp.core.property.PropertyService : Inner update property, key=ocp.agent.version.full, previousValue=4.3.5-20250319105844, updatedValue=4.3.5-20250319105844, runningValue=4.3.5-20250319105844 2025-04-08 21:03:04.315 INFO 26087 --- [main,,] c.o.o.s.dataclean.TableCleaningService : Successfully init all table cleaning configurations: {"cleanStatusTransition":{"batchIntervalMillis":0,"batchSize":0,"condition":"update_time <= date_sub(curdate(), interval 30 day)","priority":0,"tableName":"ops_status_transition","timeoutNs":0},"cleanDistributedEvent":{"batchIntervalMillis":0,"batchSize":0,"condition":"expire_time <= date_sub(curdate(), interval 30 day)","priority":1,"tableName":"distributed_event","timeoutNs":0},"cleanDistributedEventReception":{"batchIntervalMillis":0,"batchSize":0,"condition":"event_id in (select id from distributed_event where expire_time <= date_sub(curdate(), interval 30 day))","priority":0,"tableName":"distributed_event_reception","timeoutNs":0},"cleanStorageObjectBlock":{"batchIntervalMillis":10000,"batchSize":10,"condition":"file_id not in (select id from storage_object_meta)","priority":0,"tableName":"storage_object_block","timeoutNs":0}} 2025-04-08 21:03:04.738 INFO 26087 --- [main,,] f.a.AutowiredAnnotationBeanPostProcessor : Autowired annotation is not supported on static methods: public static void com.oceanbase.ocp.core.util.jpa.JsonNodeConverter.init(com.fasterxml.jackson.databind.ObjectMapper) 2025-04-08 21:03:04.978 INFO 26087 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup daemon submitted 2025-04-08 21:03:04.980 INFO 26087 --- [main,,] s.i.c.SqlAuditRawStatRollupDaemonService : SQL stat rollup workers submitted 2025-04-08 21:03:05.063 WARN 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : Error state, the follower found itself leader. 2025-04-08 21:03:05.063 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :null 2025-04-08 21:03:06.225 INFO 26087 --- [main,,] c.o.ocp.ha.maz.OcpAzoneDaemonService : Multi azone daemon not enabled 2025-04-08 21:03:06.324 INFO 26087 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init start... 2025-04-08 21:03:06.355 INFO 26087 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:06.356 INFO 26087 --- [main,,] c.o.ocp.monitor.OcpMonitorManager : OcpMonitorManager init success. 2025-04-08 21:03:06.398 INFO 26087 --- [main,,] c.o.o.m.c.MonitorDataRolloverProperties : Monitor retention conf is {ob_cluster_system_event=31, ob_cluster_server_event=31, ocp_metric_data_1=8, ocp_metric_data_60=31, metric_data_async=1, metric_hour_data=1098, metric_daily_data=1098} 2025-04-08 21:03:06.593 INFO 26087 --- [main,,] .r.DefaultUserResourceManagerServiceImpl : not UPGRADE, skip create default user resource 2025-04-08 21:03:06.836 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_config_template.yaml in jar 2025-04-08 21:03:06.843 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_agent_ob_relations_config.yaml in jar 2025-04-08 21:03:06.845 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:03:06.846 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupDataConfiguration : read from physical_backup_ability_ob_version_relation_config.yaml in jar 2025-04-08 21:03:06.846 INFO 26087 --- [main,,] c.o.o.b.i.c.BackupConfiguration : init backup agent meta cfg 2025-04-08 21:03:06.869 INFO 26087 --- [main,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key global_lock_com.oceanbase.ocp.backup.internal.task.backupocptaskmanager 2025-04-08 21:03:06.874 INFO 26087 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : upgradeFromOldAgentToNewService start. 2025-04-08 21:03:06.919 INFO 26087 --- [main,,] c.o.o.b.i.a.BackupRestoreAgentService : serviceInfos=[], allMachineList=[] 2025-04-08 21:03:06.935 INFO 26087 --- [main,,] c.o.o.b.i.task.BackupOcpTaskManager : register backup schedules success 2025-04-08 21:03:06.985 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=CMDB_SYNCER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:31+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:06.987 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=MC_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:31+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:06.989 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PERF_SQL_ROLLUP, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:21+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:07.001 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:21+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:07.016 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=RATE_LIMIT, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:31+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:07.028 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:21+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:07.030 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:21+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:07.032 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=TASK_MANAGER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:01:21+08:00, updateTime=2025-04-08T21:01:31+08:00, expireTime=null, isLeader=false} 2025-04-08 21:03:07.033 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.038 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service MC_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.042 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PERF_SQL_ROLLUP, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.057 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service CMDB_SYNCER, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.062 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service TASK_MANAGER, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.064 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service RATE_LIMIT, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.083 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.085 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 21:03:07.303 INFO 26087 --- [main,,] c.o.ocp.alarm.AlarmBeanConfiguration : initialize notificationSendExecutor 2025-04-08 21:03:07.449 INFO 26087 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Slice modified, module=metric-collect, name=192.168.1.61:8080, consumerId=3, resources=[ShardingSlice(consumerId=3, type=OB_CLUSTER, resourceIds=[2]), ShardingSlice(consumerId=3, type=HOST_MONITOR, resourceIds=[2])] 2025-04-08 21:03:07.454 INFO 26087 --- [sharding-metric-collect1,,] c.o.o.core.sharding.ShardingCoordinator : Start rebalance, consumerSize=1, sliceEntitySize=2, strategy=com.oceanbase.ocp.monitor.sharding.MonitorShardingStrategy@4d3a732b 2025-04-08 21:03:07.454 INFO 26087 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=PAUSE 2025-04-08 21:03:07.961 INFO 26087 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=STATELESS. 2025-04-08 21:03:07.976 INFO 26087 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Internal auth configured. 2025-04-08 21:03:07.977 INFO 26087 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Access key auth configured. 2025-04-08 21:03:07.981 INFO 26087 --- [main,,] s.c.c.s.c.StatelessSecurityConfiguration : Basic auth enabled. 2025-04-08 21:03:08.064 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 gets the leadership... 2025-04-08 21:03:08.319 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure com.oceanbase.ocp.server.common.config.security.conf.StatelessSecurityConfiguration$$Lambda$2003/1356806002@6ae375bf with [org.springframework.security.web.session.DisableEncodeUrlFilter@77c18723, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@749df68b, org.springframework.security.web.context.SecurityContextPersistenceFilter@94f54fe, org.springframework.security.web.header.HeaderWriterFilter@3f2ff817, org.springframework.security.web.authentication.logout.LogoutFilter@64698769, com.oceanbase.ocp.server.common.config.security.accesskey.ContentCacheRequestFilter@13df5bed, com.oceanbase.ocp.server.common.config.security.filter.InternalAuthenticationFilter@62e08aa4, com.oceanbase.ocp.server.common.config.security.accesskey.AccessKeyAuthenticationFilter@1963f144, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@251654d, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@8a6355b, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@64c2a0e7, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@29f765c4, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@6b3846d9, org.springframework.security.web.session.SessionManagementFilter@75f7d7b1, org.springframework.security.web.access.ExceptionTranslationFilter@65729cba, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@390761d3] 2025-04-08 21:03:08.329 INFO 26087 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : storage exporter register: collectEnable=true, periodMinutes=180 2025-04-08 21:03:08.329 INFO 26087 --- [main,,] c.o.o.b.i.m.s.StorageExporterServiceImpl : register backup exporter, periodMinutes=180 2025-04-08 21:03:08.522 INFO 26087 --- [main,,] o.o.core.config.InitializationService : Initializing OpenSAML using the Java Services API 2025-04-08 21:03:08.656 INFO 26087 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmlenc#ripemd160 2025-04-08 21:03:08.663 INFO 26087 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#hmac-ripemd160 2025-04-08 21:03:08.674 INFO 26087 --- [main,,] o.o.xmlsec.algorithm.AlgorithmRegistry : Algorithm failed runtime support check, will not be usable: http://www.w3.org/2001/04/xmldsig-more#rsa-ripemd160 2025-04-08 21:03:09.797 INFO 26087 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : Configure session, policy=IF_REQUIRED. 2025-04-08 21:03:09.798 INFO 26087 --- [main,,] .s.c.c.s.c.OcpSecurityConfiguratorHelper : CSRF configured. 2025-04-08 21:03:10.144 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@2a5d4308, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@d88180f, org.springframework.security.web.context.SecurityContextPersistenceFilter@ee81f7c, org.springframework.security.web.header.HeaderWriterFilter@4c9ed2ca, org.springframework.security.web.csrf.CsrfFilter@103e1cf0, org.springframework.security.web.authentication.logout.LogoutFilter@6538889c, org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter@663e8537, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter@440c9bc7, com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter@191de414, org.springframework.security.oauth2.client.web.OAuth2LoginAuthenticationFilter@3ab367bd, org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationFilter@2ae7ccd6, com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter@4cc9b182, com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter@c4cee10, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@481c044d, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@1119c7e, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@3b230614, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@4149c7f6, org.springframework.security.web.session.SessionManagementFilter@2c278e35, org.springframework.security.web.access.ExceptionTranslationFilter@1213bcbb, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@2171bbf0] 2025-04-08 21:03:10.180 INFO 26087 --- [main,,] c.o.o.s.c.config.SpringScheduleConfig : Spring scheduled pool size is 30. 2025-04-08 21:03:10.451 INFO 26087 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from PAUSE to WAITING. 2025-04-08 21:03:10.531 INFO 26087 --- [sharding-metric-collect1,,] c.o.o.c.sharding.registry.ShardingTask : Mark consumer state, newState=ASSIGNED 2025-04-08 21:03:10.546 INFO 26087 --- [main,,] c.o.o.s.o.s.c.OverviewBeanConfiguration : initialize overviewContainerExecutor 2025-04-08 21:03:10.870 WARN 26087 --- [main,,] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning 2025-04-08 21:03:11.064 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:11.072 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:11.766 INFO 26087 --- [main,,] o.s.b.a.w.s.WelcomePageHandlerMapping : Adding welcome page: class path resource [static/index.html] 2025-04-08 21:03:12.355 INFO 26087 --- [prometheus-minute-schedule-1,edf4f61c1527c4f6,de9cf35d7f6a66a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:13.475 INFO 26087 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.c.s.registry.ConsumerRegistry : Modify consumer state, from ASSIGNED to WORKING. 2025-04-08 21:03:13.475 INFO 26087 --- [heartbeat-metric-collect-192.168.1.61:80801,,] c.o.o.s.c.config.MetricConfiguration : Consumer had rebalanced. 2025-04-08 21:03:14.072 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:14.073 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:15.203 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/assets/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.203 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/assets/**'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/index.html']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/index.html'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.js']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.js'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.css']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.css'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/webjars/**'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.svg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.svg'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/login']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/login'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/docs/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/docs/**'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/favicon.ico']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/favicon.ico'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/static/**']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/static/**'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.woff2']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.woff2'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.ttf']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.ttf'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.tif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.tif'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/**/*.gif']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/**/*.gif'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.jpg']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.jpg'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='**/*.png']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='**/*.png'] 2025-04-08 21:03:15.204 WARN 26087 --- [main,,] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/pre_login_loading']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. 2025-04-08 21:03:15.204 INFO 26087 --- [main,,] o.s.s.web.DefaultSecurityFilterChain : Will not secure Ant [pattern='/pre_login_loading'] 2025-04-08 21:03:15.355 INFO 26087 --- [prometheus-minute-schedule-1,9aa2d0cc4ae11d6e,0cc0062e24a3b4eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:15.975 INFO 26087 --- [main,,] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator' 2025-04-08 21:03:16.252 INFO 26087 --- [main,,] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2025-04-08 21:03:16.481 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=15 2025-04-08 21:03:16.481 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=alarmDetectOnce, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=fixAlarmTarget, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=inhibitAlarms, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshDetectorFactory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshGlobalSilencer, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshActiveAlarmStatus, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=resolveAlarmsOfAbsentTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshBackupAlarmStat, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmSummary, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAggregators, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshHostAlarmInfo, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=removeInactiveAlarmEvents, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTaskManager, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=20, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmTargets, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAlarmInMemory, conf=ScheduledConf{initialDelay=60, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=7 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostAgentStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshAgentConfig, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=10, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=getHostAgentAlarms, conf=ScheduledConf{initialDelay=2, cron='null', fixedDelay=1, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostClockDiff, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostVersion, conf=ScheduledConf{initialDelay=30, cron='null', fixedDelay=4, fixedRate=0, timeUnit=MINUTES, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkHostStatus, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkCpuBindCore, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogClusters, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=30, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncBinlogInstances, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp multi cluster mode disabled. 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.common.scheduled.OcpSchedules : Ocp vpd mode disabled. 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=8 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanupDruidCache, conf=ScheduledConf{initialDelay=600, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkOdpClient, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=partitionDaemon, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskRecords, conf=ScheduledConf{initialDelay=0, cron='0 20 5 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=refreshOcpConfig, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=180, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectOdpMetrics, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTmpLogFiles, conf=ScheduledConf{initialDelay=0, cron='0 20 4 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=cleanTaskLog, conf=ScheduledConf{initialDelay=0, cron='0 10 3 * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=6 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllTenantServiceNameConflict, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncAllTenantInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllStandbyTenant, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMajorCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllTenantMiniCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkSuspendTenantCompactionStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=2 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkObproxyServers, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.482 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectObproxyMetrics, conf=ScheduledConf{initialDelay=120, cron='10/30 * * * * *', fixedDelay=0, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=11 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=120, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectAllClusterCompaction, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkAllArbitrationService, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterRootServer, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=300, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterCgroupValue, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationServiceGroup, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=10, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkArbitrationAutoReplaceStatusTask, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkClusterStatus, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=checkStandbyClusterInfo, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=0, fixedRate=30, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.483 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=syncClusterSystemEvent, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=600, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.501 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Ocp scheduled task specified config name, methodName=collectHighRiskSqlAuditStat, configName=collectHighSqlAuditStat 2025-04-08 21:03:16.502 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Scheduled tasks size=4 2025-04-08 21:03:16.502 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=oasSqlCuring, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.508 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectSlowSqlForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.508 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectTransStatForAlarm, conf=ScheduledConf{initialDelay=0, cron='null', fixedDelay=60, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.508 INFO 26087 --- [main,,] c.o.o.s.c.scheduled.AbstractSchedules : Add scheduled task, name=collectHighRiskSqlAuditStat, conf=ScheduledConf{initialDelay=120, cron='null', fixedDelay=300, fixedRate=0, timeUnit=SECONDS, enabled=true} 2025-04-08 21:03:16.521 INFO 26087 --- [ocp-schedules-20,1396b34076219bb5,2f49d16652f15a67] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:03:16.525 INFO 26087 --- [ocp-schedules-7,d342b5c38374101c,983157b16c354499] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:03:16.525 INFO 26087 --- [ocp-schedules-25,12349d91db7f2269,4f7319aa6c0883c2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:03:16.528 INFO 26087 --- [ocp-schedules-8,5fd617caa6531a02,06c5eb6109d6c1a0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:03:16.528 INFO 26087 --- [ocp-schedules-3,04ffe645c54d951d,478d008b60701982] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:03:16.528+08:00, clusters=[2] 2025-04-08 21:03:16.529 INFO 26087 --- [ocp-schedules-2,5c52ed9bed932ac7,6da69e9226974266] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:03:16.529+08:00, clusters=[2] 2025-04-08 21:03:16.531 INFO 26087 --- [ocp-schedules-7,d342b5c38374101c,983157b16c354499] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:16.531 INFO 26087 --- [ocp-schedules-5,3127f36525b5ba95,f8812ad65c82bdab] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:03:16.532 INFO 26087 --- [ocp-schedules-21,ab9ec939df84149d,c64cf686cbcb702d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:03:16.533 INFO 26087 --- [ocp-schedules-19,5f0d580d8c6c8521,df9f87be4adf2cc7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:03:16.533 INFO 26087 --- [ocp-schedules-4,a57ba1db827d1ddf,d6134e7b71363879] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:03:16.534 ERROR 26087 --- [ocp-schedules-19,5f0d580d8c6c8521,df9f87be4adf2cc7] com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set 2025-04-08 21:03:16.534 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:03:16.534 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:03:16.534 INFO 26087 --- [ocp-schedules-24,f66f7c4a76438e39,c668108db225acb5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:03:16.534 INFO 26087 --- [ocp-schedules-23,277cea69af3f9bb0,4ed4d404283c0259] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:03:16.534 INFO 26087 --- [ocp-schedules-19,5f0d580d8c6c8521,df9f87be4adf2cc7] com.alibaba.druid.pool.DruidDataSource : {dataSource-2} inited 2025-04-08 21:03:16.535 INFO 26087 --- [ocp-schedules-23,277cea69af3f9bb0,4ed4d404283c0259] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:16.536 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ************************* ACTIVE OCP PROPERTIES ****************************** 2025-04-08 21:03:16.536 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : Active profiles: [composite] 2025-04-08 21:03:16.537 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:03:16.537 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:03:16.539 WARN 26087 --- [ocp-schedules-25,12349d91db7f2269,4f7319aa6c0883c2] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncClusterInfo, confName=syncClusterInfo 2025-04-08 21:03:16.542 INFO 26087 --- [ocp-schedules-8,5fd617caa6531a02,06c5eb6109d6c1a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:16.544 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:03:16.547 INFO 26087 --- [ocp-schedules-14,d5363a0eac0090a7,e1b0eef3b75fb133] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:03:16.549 INFO 26087 --- [ocp-schedules-18,1d62d112ed3ee066,f280dcd36a6b9c04] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:03:16.549 INFO 26087 --- [ocp-schedules-6,9247908740ef303e,2df3c0c82314ced6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:03:16.549 INFO 26087 --- [ocp-schedules-28,d934eccf7a3bd18f,5c687a6d8d6efa06] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:03:16.549 INFO 26087 --- [ocp-schedules-22,dc983bc57af7a74f,93642f90af0be560] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:03:16.550 INFO 26087 --- [ocp-schedules-29,01a1454fc2a60066,b3005eec554eecdf] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:03:16.551 INFO 26087 --- [ocp-schedules-10,8165bcc95347a9a0,1de5be302e2ce085] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : agent.log.query.http.timeout=60s 2025-04-08 21:03:16.551 INFO 26087 --- [ocp-schedules-11,cc59bb68e3983c88,0cbf5fa03040e6ff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : backup.dbname.prefix=backup 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : binlog.cluster.parameter.blacklist= 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : display-parse-failed-sql=true 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : enable-sql.parse-service=true 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file=/root/ocp-server/log/ocp.log 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.clean-history-on-start=true 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-history=100 2025-04-08 21:03:16.551 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.max-size=100MB 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.name=/oceanbaselog/sgkyocp/ocp-server.log 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.file.total-size-cap=10GB 2025-04-08 21:03:16.552 WARN 26087 --- [ocp-schedules-11,cc59bb68e3983c88,0cbf5fa03040e6ff] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=checkClusterCgroupValue, confName=checkClusterCgroupValue 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.com.oceanbase.ocp=INFO 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.org.hibernate.SQL=INFO 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-1,6d02cfbab455485e,0ce5d2e6ce72f880] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.level.web=INFO 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.console=%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(%5p) %clr(26087){magenta} %clr(---){faint} %clr([%t,%X{traceId},%X{spanId}]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %.-2048m%n%wEx 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-29,01a1454fc2a60066,b3005eec554eecdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} %5p 26087 --- [%t,%X{traceId},%X{spanId}] %-40.40logger{39} : %.-2048m%n%wEx 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : mgragent.log.level=info 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : monagent.log.level=info 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.hidden.parameter.whitelist=__min_full_resource_pool_memory,_rowsets_max_rows,_ob_max_thread_num,_enable_defensive_check,_bloom_filter_ratio,_io_read_batch_size,_io_read_redundant_limit_percentage,_enable_kv_feature 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.enabled=true 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ob.logcleaner.run.internal=5m 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.cleanup.period.seconds=300 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.holder.capacity=2000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.executor.thread-count=16 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.init.timeout.millis=10000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.connector.max-idle.seconds=3600 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-14,d5363a0eac0090a7,e1b0eef3b75fb133] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-22,dc983bc57af7a74f,93642f90af0be560] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.ob.connection.mode=direct 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.connect.timeout.ms=10000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.jdbc.socket.timeout.ms=10000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.connect.timeout.ms=20000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.obproxy.vpc.jdbc.socket.timeout.ms=20000 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-30,a16e46a3df2d80f7,ac955d2479a4bb71] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.operation.global.timeout.millis=300000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.print.sql=true 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.query.global.timeout.millis=10000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.slow.query.threshold.millis=1000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.connect.timeout.millis=2000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : obsdk.socket.read.timeout.millis=1800000 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.anomaly.event.analyzer-sql-cpu-percent-limit=15 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.config={"enabled":true, "analyzeRangeMin" : 5, "offsetMin": 1, "cpuPercentLimit" : 90} 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp-perf.sql.cure.enabled=true 2025-04-08 21:03:16.552 WARN 26087 --- [ocp-schedules-18,1d62d112ed3ee066,f280dcd36a6b9c04] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=syncAllTenantInfo, confName=syncAllTenantInfo 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.metric-auth-enabled=true 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-6,9247908740ef303e,2df3c0c82314ced6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.auth.username=ocp_agent 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.collect.snapshot.status=active 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.force.ssh.installation=false 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.home.path.prefix=/home/admin 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-28,d934eccf7a3bd18f,5c687a6d8d6efa06] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.manager.http.port=62888 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.monitor.http.port=62889 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.config={"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80},{"file_regex":"[a-z]+.log.[0-9]+.zst","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-2,5c52ed9bed932ac7,6da69e9226974266] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.enabled=true 2025-04-08 21:03:16.552 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.ob.log-cleaner.run.internal=5m 2025-04-08 21:03:16.552 INFO 26087 --- [ocp-schedules-10,8165bcc95347a9a0,1de5be302e2ce085] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.snapshot.maxsize=1GB 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.use.specific.http.ip=false 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.agent.version.full=4.3.5-20250319105844 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.auto.resolved.alarm-types=ob_tenant_task_timeout,ob_host_task_timeout,standby_tenant_sync_delay_too_long 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.channel.modify.enabled=true 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-expired-seconds=600 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.query-slow-sql-timeout-seconds=3 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-circuit-breaker-seconds=300 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slow-sql-exclude-obclusters=sys#* 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.slowsql-query-limit=2000 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-duration-seconds=60 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-min-log-size-mb=0.5 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-query-limit=2000 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-circuit-breaker-seconds=300 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans-stat-exclude-obclusters= 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.datasource.trans.query-end-time-offset-seconds=15 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.detect.too-many-targets-alarm-threshold=20 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.enabled=true 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event-readable-cluster-auth=true 2025-04-08 21:03:16.553 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.event.resend-interval-seconds=180 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.host.property=svr_ip 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-batch-count=3000 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.archive-once-times=100 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.notification.expired-days=90 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric-offset-seconds=10 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.metric.only-cache=true 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.minute.metric-window-ratio=3 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.second.metric-window-ratio=6 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.query.sql.metric-limit=2000 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.resolve-timeout-minutes=60 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.batch-max=20 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.enabled=true 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-retry-times=1 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.once-timeout-ms=10000 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.period.ms=2000 2025-04-08 21:03:16.553 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.retry-timeout-minutes=60 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-failed-retry-times=3 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.total-timeout-seconds=60 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.alarm.send.will-be-sent-in-period-seconds=1800 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.enabled=false 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.addresses=null 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.password=null 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.client.username=null 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.keep.days=7 2025-04-08 21:03:16.554 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.replicas=1 2025-04-08 21:03:16.556 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.es.index.shards=12 2025-04-08 21:03:16.556 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.log.enabled=false 2025-04-08 21:03:16.556 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.analyze.ob.trace.enabled=false 2025-04-08 21:03:16.556 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.application.uuid=493eecb387654b6f90f5379d8459859c 2025-04-08 21:03:16.556 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.audit.enabled=true 2025-04-08 21:03:16.557 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.enabled=false 2025-04-08 21:03:16.557 WARN 26087 --- [ocp-async-6,28116f9c02095196,6f9bd804d3b58b46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:16.558 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-location= 2025-04-08 21:03:16.558 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.advanced-secondary-backup.target-cloud-name= 2025-04-08 21:03:16.558 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.agent.relation.file.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/backup_agent_ob_relations_config.yaml')} 2025-04-08 21:03:16.558 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.alarm.backup-liboblog-expire-days=7 2025-04-08 21:03:16.559 INFO 26087 --- [ocp-schedules-3,04ffe645c54d951d,478d008b60701982] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.backup-command.timeout=300 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.logical.log.meta.time.continuity.threshold.seconds=10 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.object-storage-sdk.invoke.https=true 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.obs.support.enabled=false 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.oss.support.cname=true 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.restore-time-pre-check.enabled=true 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.command.timeout=3600000 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.host.task.concurrency=8 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.max.retry.times=2 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-interval-seconds=60 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.obtain-idle-host-times=60 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.period.minutes=180 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.schedule.enabled=true 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.backup.storage.collect.task.timeout=10800000 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.build.version.check=false 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ce.supported.ob.regex=^oceanbase-actiondb-(\d+\.){2}(\d+)(\.\d+)?-([\d.]+)\.([a-zA-Z0-9]+)\.(x86_64|aarch64)(\.rpm)?$ 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.check.host.http.read.timeout=5000 2025-04-08 21:03:16.559 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.data.platform.site.url=null 2025-04-08 21:03:16.560 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cloud.meta.platform.site.url=null 2025-04-08 21:03:16.560 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.cluster.obVersion.parameter.template.relation=[{"versionList":["2.2.77"],"parameterTemplateName":"ob.cluster.parameter.template.built-in.default-2-2-77.name"}] 2025-04-08 21:03:16.564 ERROR 26087 --- [ocp-schedules-25,12349d91db7f2269,4f7319aa6c0883c2] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterInfo(ClusterSchedules.java:80) ... 15 common frames omitted 2025-04-08 21:03:16.564 ERROR 26087 --- [ocp-schedules-18,1d62d112ed3ee066,f280dcd36a6b9c04] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) ... 15 common frames omitted 2025-04-08 21:03:16.564 ERROR 26087 --- [ocp-schedules-11,cc59bb68e3983c88,0cbf5fa03040e6ff] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: java.lang.NullPointerException: null at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.adaptTimeout(AbstractSchedules.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.checkClusterCgroupValue(ClusterSchedules.java:135) ... 15 common frames omitted 2025-04-08 21:03:16.565 INFO 26087 --- [ocp-schedules-30,a16e46a3df2d80f7,ac955d2479a4bb71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:16.567 INFO 26087 --- [ocp-schedules-26,d5625f83814f2bc2,fb60343856a539b1] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 21:03:16.567 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.site.url= 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config-url.use-request-address=true 2025-04-08 21:03:16.568 INFO 26087 --- [ocp-schedules-9,da74d8079ee33713,1219106ffb538e14] c.o.o.m.service.SeriesIdCleanService : Clean series by rules started. 2025-04-08 21:03:16.568 INFO 26087 --- [ocp-schedules-1,1b820753c5fd6dea,cebd86d4100b4114] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.cache.refresh.interval=60 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.connect-timeout=2000 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.datasource.socket-timeout=30000 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.initialSize=10 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.keepAlive=true 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxActive=100 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.maxWait=2000 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.minIdle=2 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.name=config-server-connect-pool 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.testWhileIdle=true 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.config.server.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 21:03:16.568 INFO 26087 --- [ocp-schedules-4,5b908a78f6258697,eb018fac5df502f8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.expire.seconds=30 2025-04-08 21:03:16.568 INFO 26087 --- [ocp-schedules-27,525d12745c377f58,548105c9d3bb4a06] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.server.update.seconds=10 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.expire.seconds={CMDB_SYNCER:30, TASK_MANAGER:30, RATE_LIMIT:30, PERF_SQL_ROLLUP:30, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:30} 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.distributed.service.update.seconds={CMDB_SYNCER:10, TASK_MANAGER:10, RATE_LIMIT:10, PERF_SQL_ROLLUP:15, SQL_INDEX_DIAGNOSER:30, SQL_PERF_DIAGNOSER:30, PLAN_CHANGE_DETECTOR:30, MC_MANAGER:10} 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.event.snapshot.maxage-days=90 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.built-in.bucket=built-in 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.default-block-split-size=1048576 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.has.signature.keywords.list=PGP,signatures 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.built-in.dir=/root/ocp/ocp-server/lib 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.local.dir=/home/root/software 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.max-concurrent-count=16 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.oss-block-split-size=1048576 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.enabled=true 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.signature.check.invalid.keywords.list=NOT OK 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.supported-architecture=x86_64,AArch64 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.sys-package.bucket=sys-package 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.file.try-lock-timeout-milliseconds=60000 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.config.threshold={"-1":{"host_load":{"load1_per_cpu":2},"host_disk_status":{"host_disk_used_percent":85},"host_time":{"clock_diff":500,"ssh_interval":1000},"ob_server_disk_status":{"observer_disk_threshold_percentage":80},"ob_server_partition_count":{"ob_server_partition_count":100000},"obproxy_connection_count":{"obproxy_session_count_percentage":80}}} 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.available-keyword=okay 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.health-check.status.unavailable-keyword=bad 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.enable=true 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.max-diff=50 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.check.clock-diff.mode=0 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.clockdiff.timeout.millis=20000 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.default-timeout-millis=30000 2025-04-08 21:03:16.568 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.remote-command.min-timeout-millis=30000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.host.ssh-ping.max-delay-millis=3000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.account.system.active=false 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth=composite 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.access-key.enabled=true 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.basic.enabled=true 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.auth.whitelist=/api/v2/time,/api/v2/mc/info,/api/v2/sensitiveDataKey,/api/v2/obproxy/loadBalanceInfo.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|GetObRootServiceInfoUrlTemplate|ObRootServiceUnregister|GetObTenantInfoByServiceNames|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,performance/ash/reports/.*/data 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.enabled=true 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.csrf.url.excluded=/api/v2/iam/login.*,.*Action=(ObRootServiceInfo|ObRootServiceRegister|ObRootServiceUnregister|ObIDCRegionInfo|GetObProxyConfig|AllocateClusterIdByNameAndIdx|DeleteObRootServiceInfoByClusterName|RegisterObTenantInfo|ModifyTenantInfo|DeleteTenantInfo).*,/api/v2/monitor/metric/scan.* 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.lockout-minutes=1 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.max-attempts=60 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.client.white-list=null 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.lockout-minutes=30 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.login.max-attempts=5 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.preserved.username.list=admin,sys,system,sysdba,dba,oceanbase,obproxy,proxyro,ocp,ocp_monitor,ocp_agent,odc,oms,backup,drc,lbacsys,odp,temp 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.enabled=true 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.connect.timeout=1000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.internal-api.read.timeout=1000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-ip.enabled=false 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.on-user.enabled=true 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.rate-limit.task.schedule.period=3 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.role.forbidden.update.username.list=admin,system,obproxy,ocp_admin,monitor 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.sso-login-url= 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.iam.user.access-key.max.count=5 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.idempotent.client-token.expire.time=14d 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.cleanup.enabled=false 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.inspection.report.expired.days=15 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.internal.rpc.user.username=system 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.connect.timeout=180000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.http.read.timeout=180000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.download.tmp.dir=/root/ocp/logs/ocp 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.max-download-speed=1024 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.log.query.http.read.timeout=70000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.cluster.name=ocpdb 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.meta.disconnect.alarm.minutes=2 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.async-compute.enabled=true 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.metric.collect.interval.second=15 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.query.min.time.buffer.second=0 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.async.write.queue.capacity=5000000 2025-04-08 21:03:16.569 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.enabled=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.host.enabled=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.max-pool-size=3000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obcluster.enabled=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obproxy.enabled=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.obsdk.sql-query-row-limit=10000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.collect.request.timeout={"second_connect_timeout":1000,"second_read_timeout":3000,"second_request_timeout":2000,"minute_connect_timeout":10000,"minute_read_timeout":30000,"minute_request_timeout":40000} 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.custom.collectItem.sql.function.whiteList=count,time_to_usec,timestampdiff,coalesce,sum,max,avg,min,round,unix_timestamp,abs 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.retention-days={"ob_cluster_system_event":31,"ob_cluster_server_event":31,"ocp_metric_data_1":8,"ocp_metric_data_60":31,"metric_data_async":1,"metric_hour_data":1098,"metric_daily_data":1098} 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.data.table-switch-time=1744114820 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.query.timeout.sec=120 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.datasource.socket.timeout.ms=120000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.deletable.metricNames=process_count|app=HOST|name=observer,process_count|app=HOST|name=obproxy,process_cpu_percent|app=HOST|name=observer,process_cpu_percent|app=HOST|name=obproxy,process_exists|app=HOST|name=observer,process_exists|app=HOST|name=obproxy,process_fd_count|app=HOST|name=observer,process_fd_count|app=HOST|name=obproxy,process_memory_rss_bytes|app=HOST|name=observer,process_memory_rss_bytes|app=HOST|name=obproxy,process_thread_count|app=HOST|name=observer,process_thread_count|app=HOST|name=obproxy,process_uptime_seconds|app=HOST|name=observer,process_uptime_seconds|app=HOST|name=obproxy,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_system_event_detail_total_waits,ob_system_event_detail_total_timeouts,ob_system_event_detail_time_waited_us,ob_latch_get_total,ob_latch_miss_total,ob_latch_sleep_total,ob_latch_immediate_get_total,ob_latch_immediate_miss_total,ob_latch_spin_get_total,ob_latch_wait_seconds_total,ob_tenant_context_memory_hold_bytes,ob_tenant_context_memory_used_bytes,ob_tenant_context_memory_count 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-active-period-seconds=30 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.check-inactive-period-seconds=30 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.exporter.inactive.threshold.seconds=180 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.host.exporters=/metrics/node/host 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.logproxy.exporters=/metrics/logproxy 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.rollup.type=MAX 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.collect.service.type=LOCAL 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.interval.factor=2 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.metric.service.type=LOCAL 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.exporters=/metrics/ob/extra,/metrics/obcloud/extra 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.minute.write.queue.capacity=5000000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.ob.exporters=/metrics/node/ob,/metrics/ob/basic,/metrics/ob/extra 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.obproxy.exporters=/metrics/node/obproxy,/metrics/obproxy 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.prometheus.service.discovery.enabled=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.exporters=/metrics/node/host,/metrics/node/ob,/metrics/ob/basic,/metrics/node/obproxy,/metrics/obproxy,/metrics/node/oblb,/metrics/node/obdns,/metrics/logproxy 2025-04-08 21:03:16.570 INFO 26087 --- [Check status for all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_status_check', labels={app=OB, check_item=cluster connection, failed_reason=-, ob_cluster_id=1744114127, ob_cluster_name=ocpdb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744117396 : 2025-04-08 13:03:16 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.second.write.queue.capacity=5000000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitor.server.exporter.auth.enabled=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.database=monitor_database 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.driverClassName=com.oceanbase.jdbc.Driver 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.initialSize=10 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.keepAlive=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxActive=200 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.maxWait=2000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.minIdle=2 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.name=monitordb-connect-pool 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.testWhileIdle=true 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.ConnectionTimeout=2000 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.maximumPoolSize=200 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.minimumIdle=10 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.hikari.poolName=monitor-connect-pool 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.host=192.168.1.61 2025-04-08 21:03:16.570 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.password=****** 2025-04-08 21:03:16.571 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.port=2881 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.public-key=null 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.series.scan.batch-size=2000 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.monitordb.username=root@ocp_monitor 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.multicloud.monitordb.password-secret-key=****** 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.agent.socks-port=1080 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.host-address=localhost 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.socks-port=1080 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.inter-connector.web-port=8088 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.network.vpc.mode.enabled=false 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.batch.replace.arbitration.thread.core-pool-size=100 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.cluster.ops.blacklist=ocpdb 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.database.name.pattern=^[a-zA-Z_0-9-]{2,64}$ 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.enable=false 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-core-size=20 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.resource.cache.refresh.thread-max-size=200 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.slowsql.threshold=100000 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.standby.replication.role.privilege.objects=DBA_OB_TENANTS,DBA_OB_LS_HISTORY,GV$OB_UNITS,GV$OB_PARAMETERS,GV$OB_LOG_STAT,DBA_OB_LS,DBA_OB_ACCESS_POINT 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.switchover.check.to.primary.wait-period-seconds=3 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.allow-downgrade=false 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.clean-tmp-rpm=true 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.dependencies.full-path.name=#{systemProperties['user.home'].concat('/ocp-server/etc/oceanbase_upgrade_dep.yml')} 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.ob.upgrade.script.cmd.timeout=0 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obcloud.api.address= 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obproxy.query-with-ob-connections=true 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.obshell.port=2886 2025-04-08 21:03:16.572 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.cgroup.minimum-ob-version=4.0.0.0 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.default.os.user=admin 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.connector.cache.max-idle-seconds=3600 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connect-timeout=5000 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-per-route=100 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.connection.max-pool-size=100 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.read-timeout=30000 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.socket-timeout=3000 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.core-pool-size=500 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.keep-alive-time=120000 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.http.thread.max-pool-size=1500 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.connect-timeout=10000 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.executor.ssh.default-command-timeout=10000 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.delete-server.reserve-data-enabled=false 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.host.arm-lse-supported=true 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.resource.trend.max-point=1440 2025-04-08 21:03:16.573 INFO 26087 --- [ocp-schedules-26,d5625f83814f2bc2,fb60343856a539b1] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.server.skip-io-bench=false 2025-04-08 21:03:16.573 INFO 26087 --- [ocp-schedules-26,d5625f83814f2bc2,fb60343856a539b1] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=0 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.single-idc-for-zone-enabled=true 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.enabled=false 2025-04-08 21:03:16.573 INFO 26087 --- [ocp-schedules-26,d5625f83814f2bc2,fb60343856a539b1] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=0 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.sql-execute.sql-type.whitelist=SELECT,SHOW 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.tenant.whitelist.attach-default-whitelist=true 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.ob.unit.unused.max-reserve-hour=1 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.prohibit.obsharding.operation=true 2025-04-08 21:03:16.573 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.password=****** 2025-04-08 21:03:16.574 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.readonly.default.username=proxyro 2025-04-08 21:03:16.575 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.obproxy.skip-set-proxyro-password=****** 2025-04-08 21:03:16.575 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.operation.odp.root.default.password=****** 2025-04-08 21:03:16.575 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.alarm.status.refresh.interval=10000 2025-04-08 21:03:16.575 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.backup.status.refresh.interval=300000 2025-04-08 21:03:16.575 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.cluster.status.refresh.interval=10000 2025-04-08 21:03:16.575 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.host.status.refresh.interval=10000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.obproxy.status.refresh.interval=10000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.performance.status.refresh.interval=300000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tasks.status.refresh.interval=10000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.overview.tenant.status.refresh.interval=10000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.earliest-analyse-time=8 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.max-analyse-range=60m 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ash.report-retention-days=90 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.log.threshold.byte=524288 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.big.transaction.time.threshold.ms=500 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.collect.metadb.enabled=false 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.count-sql.slow-sql-explicitly=true 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.oas.stat.enabled=true 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.password=****** 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.ocp.monitor.public.key=****** 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.limit=2000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.parallel.sql.partition.retention.days=15 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable=0 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.enable.in.147x-meta-cluster=false 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.snapshot.generation.white-list=null 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.awful-performance-index-used-config={"enabled":true, "cpuTimeLimitUs":100000, "execPsLimit":5.0, "fullLogicalReadsLimit":1000} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.cpu-time-proportion-high-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":10.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":30, "sqlCountLimit":20, "cpuTimePercentLimit":20} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.database-white-list=oceanbase,information_schema,mysql,__recyclebin,sys,__public 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.diagnose.standby-cluster.enabled=False 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.execution-spike-config={"enabled":true, "cpuTimeLimitUs":3000, "execPsLimit":5.0, "pointsLimit":5, "compareMin":20, "avgExecutionMultiply":{"0":6, "1":5, "5":4, "10":3, "50": 2.5, "100": 2, "500": 1.6}, "stdExecutionMultiply":{"0":80, "1":60 , "3": 30,"5":20, "10":15, "20":8, "30":6, "50":5, "80":4}} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.index-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.ineffective-hint-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.outline-history-query-days=30 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.perf-diagnoser-config={"enabled":true,"schedulePeriodMin":5, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":10000,"diagPeriodSec":300,"diagOffsetSec":60,"maxDiagPeriodSec":1800} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-after-plan-changed-config={"enabled":true, "compareMin":5, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2},"alarm":true} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.performance-degradation-config={"enabled":true, "cpuTimeLimitUs":10000, "execPsLimit":5.0, "maxCpuTimeLimitUs":30000, "affectedRowsLimit":100, "execLimit":5, "compareMin":20, "cpuTimeMultiply":{"0":50, "1":20, "10":10, "100":3, "1000":2}, "alarm":true, "sysTenantEnabled":false} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.query-timeout=30000000 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.row-lock-contention-high-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":100000, "elapsedTimeLimitUs":0} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.sql-diagnosis-retention-days=30 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-exists-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 21:03:16.576 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.table-scan-index-not-used-config={"enabled":true, "execPsLimit":5.0, "cpuTimeLimitUs":20000} 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql-diag.tenant-white-list=null 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.advisor-plan-max-query-range=7d 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag-plan-change-config={"enabled":true,"schedulePeriodMin":2, "coreThreadSize":8, "maxThreadSize":16,"maxQueueSize":500,"diagPeriodSec":120,"diagOffsetSec":120,"maxDiagPeriodSec":600} 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.affected-rows-threshold=50000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.core-pool-size=30 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-period-timeUs=1800000000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-pool-size=300 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-queue-size=1000000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.max-thread-pool-task-limit=50000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.partition-count-threshold=2000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.high-risk-task.return-rows-threshold=50000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-queue-max-size=1000000 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-core-size=30 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.diag.sql-audit-stat-worker-max-size=300 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.earliest-query-time=8d 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-inspect-service.enable=True 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.high-risk-max-query-range=7d 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-cpu-time-threshold=10 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.index-advisor-elapsed-time-threshold=20 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parameterize.sql=false 2025-04-08 21:03:16.577 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.info.parse.sql.type=true 2025-04-08 21:03:16.578 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.max-query-range=24h 2025-04-08 21:03:16.578 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.limit=2000 2025-04-08 21:03:16.578 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.compare.interval.days=7 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.new-sql.query.interval.days=7d 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.rows.limit=1000000 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table.limit=32 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.oas.schema-advice.table4.limit=20 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.batch.insert.size=100 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.core.thread=8 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.insert.query.rate.limit=10 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.max.thread=16 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.query.limit=2000 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.parse.queue.size=20000 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-granularity=30s 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-query-interval=2h 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.plan-hist-level0-retention=5d 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.prepare-partition-ahead=8 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-limit=0 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-digest-trend.points=15 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query-timeout=30000000 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.query.high-risk-sql.sql-text-short-length=50 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.slow-sql-limit=2000 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-audit-sample-retention=7d 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-granularity=30s 2025-04-08 21:03:16.579 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-query-interval=2h 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level0-retention=2d 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-granularity=2m 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-query-interval=12h 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level1-retention=8d 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-granularity=10m 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-query-interval=48h 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-hist-level2-retention=15d 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-info-secondary-cache-expiration=10m 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level0-query-interval=30m 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-query-digest-level1-query-interval=2h 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled=false 2025-04-08 21:03:16.583 INFO 26087 --- [ocp-schedules-12,c936b8bd4eb9dce1,6fa23bdb08c2bf61] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":0,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 21:03:16.583 INFO 26087 --- [ocp-schedules-12,c936b8bd4eb9dce1,6fa23bdb08c2bf61] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{} 2025-04-08 21:03:16.580 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.sql-stat-rollup-worker-disabled.in.147x-meta-cluster=true 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql-limit=2000 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.top-sql.query-parallel=0 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-multiply-of-plan-comparison={"0":10, "1":5, "10":2, "100":1.5, "1000":1.2} 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.sql.tuning-advisor-cpu-time-ratio-of-plan-comparison=0.8 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.detail.query.range.minute=30 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.interval.second=5 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.long.transaction.threshold.second=60 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.partition.retention.days=7 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.pending.transaction.threshold.second=60 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.query.sql.sample.timeout.second=30 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.transaction.sql.count.limit=10000 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.generate-report-thread-count=5 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.perf.workload.report.tenant-top-sql-limit=10 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.from-header=false 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.request.client.ip.header-candidates=X-Forwarded-For 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.private-key=****** 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.sensitive-data.encryption.public-key=****** 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.event.included={sql: {}, daily_merge:{}, election:{}, physical_restore:{}, observice:{}, observer:{}, clog_restore:{}, warmup:{}, log_archive:{}, archive_log:{}, minor_merge:{}, ddl:{"create_partition", "batch_create_partition", "add_replica", "remove_member", "batch_remove_member", "batch_modify_quorum", "batch_change_replica", "migrate_replica", "remove_replica", "rebuild_replica"}, freeze:{"do minor freeze fail"}, storage:{"add_replica failed", "migrate_replica failed", "reubild_replica failed", "change_replica failed", "log is new enough, turn off rebuild flag", "remove_replica failed", "restore follower replica failed", "restore_replica failed", "batch change_replica failed", "copy global index failed", "restore_standby_replica failed", "rebuild_replica failed"}} 2025-04-08 21:03:16.584 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.server.readonly.mode.enabled=false 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.site.url=http://192.168.1.61 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.standby.tenant.delay.threshold=5000 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.metadb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.startup-check.monitordb.variable=[{"variableName": "max_allowed_packet", "operator": ">=", "targetValue": 4194304}, {"variableName": "ob_read_consistency", "operator": "=", "targetValue": "STRONG"}] 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.minimum-timeout-secs=-1 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.subtask.prohibit.skip.names=[] 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.language=zh-CN 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.default.timezone= 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.event.excluded={balancer: {"start_sql_bkgd_task","finish_sql_bkgd_task"}, schema_recycler: {}} 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.cluster-name=obdv1 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.database=obproxy 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.enabled=false 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.sensitive-info={"configuredInfo":{"antmeta":{"root@sys":"9f6e186723de2f4873cad637e0e8b6e9"},"obdv1":{"root@obproxy":"8f3bab5c8face32f3a7da34de0773c9b"}},"defaultInfo":{"default":"899f7ad059fc3a3c19dee85886a8f45a7e67dd4896dbeb93"}} 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.system.obproxy.metadb.user=root@obproxy 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-interval-millis=10 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.batch-size=1000 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.preserve-days=30 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.core-pool-size=4 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.table-cleaning.thread.maximum-pool-size=16 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tag.usage.max.count=5000 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.prohibit.skip.names=["Upgrade OB cluster"] 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.fixedDelay=10 2025-04-08 21:03:16.585 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.alarmDetectOnce.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.fixAlarmTarget.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.inhibitAlarms.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshActiveAlarmStatus.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.fixedDelay=10 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAggregators.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmInMemory.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmSummary.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTargets.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.fixedDelay=20 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshAlarmTaskManager.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.fixedDelay=180 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshBackupAlarmStat.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshDetectorFactory.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.fixedDelay=10 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshGlobalSilencer.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.fixedDelay=600 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.refreshHostAlarmInfo.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.fixedDelay=600 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.removeInactiveAlarmEvents.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.fixedDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.alarm.resolveAlarmsOfAbsentTargets.initialDelay=60 2025-04-08 21:03:16.586 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogClusters.fixedDelay=30 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.binlog.syncBinlogInstances.fixedDelay=60 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.fixedRate=30 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkCpuBindCore.initialDelay=120 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.fixedRate=30 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostAgentStatus.initialDelay=120 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.fixedRate=30 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostClockDiff.initialDelay=120 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.fixedRate=30 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostStatus.initialDelay=120 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.fixedDelay=4 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.initialDelay=30 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.checkHostVersion.timeUnit=MINUTES 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.fixedDelay=1 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.initialDelay=2 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.getHostAgentAlarms.timeUnit=MINUTES 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.fixedDelay=10 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.initialDelay=2 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.host.refreshAgentConfig.timeUnit=MINUTES 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkOdpClient.fixedDelay=60 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.checkVpcConnect.fixedDelay=5 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskLog.cron=0 10 3 * * * 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTaskRecords.cron=0 20 5 * * * 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanTmpLogFiles.cron=0 20 4 * * * 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.fixedDelay=300 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.cleanupDruidCache.initialDelay=600 2025-04-08 21:03:16.587 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.collectOdpMetrics.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.ocpReplicationDaemon.fixedDelay=5 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.fixedDelay=600 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.partitionDaemon.initialDelay=120 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.local.refreshOcpConfig.fixedDelay=180 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkAllArbitrationService.fixedRate=30 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationAutoReplaceStatusTask.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkArbitrationServiceGroup.fixedDelay=10 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterCgroupValue.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterRootServer.fixedRate=300 2025-04-08 21:03:16.588 INFO 26087 --- [ocp-schedules-16,0c20d121468c00ba,6188d0f1a9578136] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkClusterStatus.fixedRate=30 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.checkStandbyClusterInfo.fixedRate=30 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.collectAllClusterCompaction.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterInfo.fixedRate=30 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncClusterSystemEvent.fixedDelay=600 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obCluster.syncRootServer.fixedDelay=120 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllStandbyTenant.fixedRate=30 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkAllTenantServiceNameConflict.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.checkSuspendTenantCompactionStatus.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMajorCompaction.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.collectAllTenantMiniCompaction.fixedDelay=300 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obTenant.syncAllTenantInfo.fixedDelay=60 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.cron=10/30 * * * * * 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.checkObproxyServers.initialDelay=120 2025-04-08 21:03:16.588 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.cron=10/30 * * * * * 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.obproxy.collectObproxyMetrics.initialDelay=120 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.fixedDelay=300 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectHighSqlAuditStat.initialDelay=120 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectSlowSqlForAlarm.fixedDelay=60 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.collectTransStatForAlarm.fixedDelay=60 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.perf.oasSqlCuring.fixedDelay=60 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.task.scheduled.pool.coreSize=30 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.mysql=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.tenant.username.pattern.oracle=^[a-zA-Z_0-9-]{2,30}$ 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version=4.3.5 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.version.full=4.3.5-20250319105844 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.context-path=/ 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ocp.web.tracking.enabled=1 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.automatic-match-work-thread=false 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.enable-strict-kernel-release=false 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.log-dir-size-threshold=1G 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.proxy-mem-limited=1G 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.syslog-level=INFO 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : odp.client.work-thread-num=8 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.elapsed-time-increase-limit=0.2 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.error-percentage-increase-limit=0.1 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.executions-increase-limit=0.2 2025-04-08 21:03:16.589 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.new-in-top-n-limit=100 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.remote-plan-percentage-increase-limit=0.1 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : perf.sql.top-sql-comparison.return-rows-increase-limit=0.2 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.address=0.0.0.0 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.port=8080 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.name=SESSION 2025-04-08 21:03:16.590 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.same-site=null 2025-04-08 21:03:16.590 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:03:16.590 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.cookie.secure=null 2025-04-08 21:03:16.590 INFO 26087 --- [ocp-schedules-9,da74d8079ee33713,1219106ffb538e14] c.o.o.m.service.SeriesIdCleanService : HostIpSize=1, clusterSize=1, obproxyClusterSize=0 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.servlet.session.timeout=30m 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.enabled=false 2025-04-08 21:03:16.590 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-alias=null 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-password=null 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store= 2025-04-08 21:03:16.590 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-password=****** 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : server.ssl.key-store-type=PKCS12 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : slow-sql.query-sql-info-from.cache.directly=false 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.max-page-size=5000 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.data.web.pageable.one-indexed-parameters=true 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.initialSize=10 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.keepAlive=true 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxActive=200 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.maxWait=2000 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.minIdle=2 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.name=metadb-connect-pool 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.testWhileIdle=true 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.druid.validationQuery=SELECT 1 FROM DUAL 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.ConnectionTimeout=2000 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.connectionTestQuery=SELECT 1 FROM DUAL 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.maximumPoolSize=200 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.minimumIdle=10 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.datasource.hikari.poolName=metadb-connect-pool 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.jpa.properties.hibernate.globally_quoted_identifiers=true 2025-04-08 21:03:16.591 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.enabled=true 2025-04-08 21:03:16.591 INFO 26087 --- [ocp-schedules-6,9247908740ef303e,2df3c0c82314ced6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:03:16.594 INFO 26087 --- [ocp-schedules-6,9247908740ef303e,2df3c0c82314ced6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:03:16.595 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-file-size=2048MB 2025-04-08 21:03:16.597 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : spring.servlet.multipart.max-request-size=2048MB 2025-04-08 21:03:16.597 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.enabled=false 2025-04-08 21:03:16.605 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.api-docs.path=/doc/api-docs 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-exclude=null 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.paths-to-match=/api/v2/** 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : springdoc.swagger-ui.path=/doc/swagger-ui.html 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : sql-stat.flush-sql-info-to-cache=true 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.core-pool-size=8 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.max-pool-size=64 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : storage.capacity.compute.executor.task-queue-size=1800 2025-04-08 21:03:16.609 INFO 26087 --- [main,,] c.o.o.s.common.trace.PropertyLogger : ****************************************************************************** 2025-04-08 21:03:16.612 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:03:16.612 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:03:16.612 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:03:16.612 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:03:16.615 INFO 26087 --- [ocp-schedules-13,3b743410e3f938d5,2c906ea8315adbe7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:03:16.615 INFO 26087 --- [ocp-async-4,5fb33d37678d898f,305638f984c60479] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:03:16.615 INFO 26087 --- [ocp-async-5,f807f14936badf24,7f31e85e1231ad7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:16.625 INFO 26087 --- [ocp-schedules-5,3127f36525b5ba95,f8812ad65c82bdab] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:03:16.628 INFO 26087 --- [main,,] c.oceanbase.ocp.OcpCeServerApplication : Started OcpCeServerApplication in 101.201 seconds (JVM running for 102.001) 2025-04-08 21:03:16.629 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:03:16.629 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:03:16.632 INFO 26087 --- [ocp-schedules-16,0c20d121468c00ba,6188d0f1a9578136] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:03:16.633 INFO 26087 --- [ocp-schedules-19,5f0d580d8c6c8521,df9f87be4adf2cc7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:03:16.633 INFO 26087 --- [ocp-schedules-19,5f0d580d8c6c8521,df9f87be4adf2cc7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:03:16.637 INFO 26087 --- [ocp-schedules-13,3b743410e3f938d5,2c906ea8315adbe7] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:03:16.657 INFO 26087 --- [pool-21-thread-1,d934eccf7a3bd18f,eed824a5730452e7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:03:16.658 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:03:16.672 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:03:16.672 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:03:16.673 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:03:16.673 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:03:16.673 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:03:16.674 INFO 26087 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : Setup sensitive data utils, publicKey=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2tCc0atuv+VPfEskkJtnsYSZP1lXZ0QdtWhag4/kNhFYQVvLKXenno/8TQo96YRBI0t2OJfdJlnaJov4HuOQ0Y7mu/FkyDs5HYx/bMHfaOGmnBGaRbEwq1ZmIHO8lgtzCl5g6vMgbSRAqjvm4o32qsRSExtqiyGmo3QhuJtErIwIDAQAB 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=profile_credential_sharing 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Metadb, table=ocp_cache_info 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_0 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_1 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat_2 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_stat 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sqltext 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_dynamic_sql 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_high_risk_sql 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_stat_0 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_plan_cache_plan_explain 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_audit_sample_statistics 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis_task 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_perf_sql_diagnosis 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_active_session_stat_0 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_snapshot 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cluster 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_tenant 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_server 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_replica 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sysstat 2025-04-08 21:03:16.676 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_row_lock 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_latch 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_cache 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_parameter 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_rpc 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_packet 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_clog 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_location_cache 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_gts 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_system_event 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_memory 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_hit 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_sql_plan_monitor 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_trans_stat_v2 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_system_event 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_cluster_server_event 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_1 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ocp_metric_data_60 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_hour_data 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_daily_data 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_second 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_minute 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_hour 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_daily 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=metric_data_async 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=sql_aggregate_data 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_hist_partition_stats_compressed 2025-04-08 21:03:16.677 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167001] Loading policy, scope=Monitordb, table=ob_tenant_mini_compaction_info 2025-04-08 21:03:16.679 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:03:16.691 INFO 26087 --- [pool-overview-container-7,864802ac622c2c12,191815cd9a7c5aae] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:03:16.694 INFO 26087 --- [pool-overview-container-8,f93c77ac578379fd,7e23f648c3918cb8] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:03:16.737 INFO 26087 --- [main,,] c.o.o.s.p.PartitionRolloverService : [OCP167004] Submitting initial worker 2025-04-08 21:03:16.742 INFO 26087 --- [ocp-schedules-9,da74d8079ee33713,1219106ffb538e14] c.o.o.m.service.SeriesIdCleanService : Clean series by rules finished. 2025-04-08 21:03:16.742 INFO 26087 --- [ocp-schedules-9,da74d8079ee33713,1219106ffb538e14] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=403 2025-04-08 21:03:16.742 INFO 26087 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP application uuid is 493eecb387654b6f90f5379d8459859c. 2025-04-08 21:03:16.742 INFO 26087 --- [main,,] c.o.o.s.c.analyzer.OcpPostStartupRunner : OCP boot ok! 2025-04-08 21:03:16.743 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=423, maxSyncedRecordId=766200877631795200 2025-04-08 21:03:16.743 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:03:16.743 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:03:16.744 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1M41.32621648S 2025-04-08 21:03:16.744 INFO 26087 --- [main,,] c.o.o.b.spring.BootstrapRunListener : ocp is ready! 2025-04-08 21:03:16.747 INFO 26087 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : ReinstallAgentAfterUpgrade created 2025-04-08 21:03:16.747 INFO 26087 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : SetProxyIdAfterUpgrade created. 2025-04-08 21:03:16.757 ERROR 26087 --- [ocp-schedules-30,a16e46a3df2d80f7,ac955d2479a4bb71] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:03:16.758 ERROR 26087 --- [ocp-schedules-29,01a1454fc2a60066,b3005eec554eecdf] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:03:16.760 INFO 26087 --- [ocp-schedules-30,a16e46a3df2d80f7,ac955d2479a4bb71] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 21:03:16.766 INFO 26087 --- [ocp-async-2,e19152ecf66b5ed1,d3eabf4b598f1c30] c.o.o.p.a.i.SessionEventSyncService : start to sync session wait event info, cluster ids is [2] 2025-04-08 21:03:16.767 INFO 26087 --- [ocp-schedules-29,01a1454fc2a60066,b3005eec554eecdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:03:16.772 ERROR 26087 --- [ocp-async-6,6e468dac4bf27557,fe6681969d459d50] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [task not exist., 999989] 2025-04-08 21:03:16.772 INFO 26087 --- [ocp-async-6,6e468dac4bf27557,fe6681969d459d50] c.o.o.c.t.e.m.TaskDefinitionManagerImpl : Task not found, task id :999989 2025-04-08 21:03:16.773 INFO 26087 --- [ocp-schedules-15,abbc80b69272a94b,e440f6cc83779741] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=266, maxSyncedRecordId=766198483195916288 2025-04-08 21:03:16.774 WARN 26087 --- [main,,] c.o.o.o.i.ObproxyOperationServiceImpl : No obproxy needs to set proxy_id, input param is empty. 2025-04-08 21:03:16.774 INFO 26087 --- [main,,] c.o.o.b.ext.SetProxyIdAfterUpgrade : Submit task null success, assign proxy_id for obproxies []. 2025-04-08 21:03:16.774 INFO 26087 --- [main,,] c.o.o.b.ext.ReinstallAgentAfterUpgrade : not UPGRADE, skip reinstall agent 2025-04-08 21:03:16.776 INFO 26087 --- [ocp-async-6,6e468dac4bf27557,fe6681969d459d50] c.o.o.s.c.l.ApplicationStartedListener : Begin to init proxyro password config in sys tenant 2025-04-08 21:03:16.777 INFO 26087 --- [ocp-schedules-9,da74d8079ee33713,1219106ffb538e14] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=423 2025-04-08 21:03:16.784 INFO 26087 --- [ocp-async-6,6e468dac4bf27557,fe6681969d459d50] c.o.o.s.c.l.ApplicationStartedListener : Success to init proxyro password config in sys tenant 2025-04-08 21:03:16.786 WARN 26087 --- [ocp-async-2,e19152ecf66b5ed1,d3eabf4b598f1c30] c.o.o.p.a.i.SessionEventSyncService : Fail to refresh cluster event info, cluster id = 2 2025-04-08 21:03:16.786 INFO 26087 --- [ocp-async-2,e19152ecf66b5ed1,d3eabf4b598f1c30] c.o.o.p.a.i.SessionEventSyncService : Sync session wait event info finished, cluster ids is [2] 2025-04-08 21:03:16.795 WARN 26087 --- [ocp-schedules-28,d934eccf7a3bd18f,5c687a6d8d6efa06] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$14662e73.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:03:16.795 INFO 26087 --- [ocp-schedules-28,d934eccf7a3bd18f,5c687a6d8d6efa06] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:03:16.816 INFO 26087 --- [pool-11-thread-1,864802ac622c2c12,d771d84121e626cf] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:03:16.816 INFO 26087 --- [pool-11-thread-1,864802ac622c2c12,d771d84121e626cf] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:03:16.816 INFO 26087 --- [pool-11-thread-2,864802ac622c2c12,1417f92b9359af79] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:03:16.819 INFO 26087 --- [pool-11-thread-2,864802ac622c2c12,1417f92b9359af79] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:03:16.821 INFO 26087 --- [pool-11-thread-3,864802ac622c2c12,e18efd6e288401a7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:03:16.825 INFO 26087 --- [pool-11-thread-4,864802ac622c2c12,5bfa94df7f3b9159] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:03:16.825 INFO 26087 --- [pool-11-thread-2,864802ac622c2c12,1417f92b9359af79] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:03:16.829 INFO 26087 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:03:16.831 INFO 26087 --- [pool-11-thread-1,864802ac622c2c12,d771d84121e626cf] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:03:16.832 INFO 26087 --- [ForkJoinPool-7-worker-9,864802ac622c2c12,07acfa6f8d8bcc38] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 21:03:16.832 INFO 26087 --- [ForkJoinPool-7-worker-9,864802ac622c2c12,07acfa6f8d8bcc38] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:03:16.832 INFO 26087 --- [ForkJoinPool-7-worker-9,864802ac622c2c12,07acfa6f8d8bcc38] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 21:03:16.833 INFO 26087 --- [pool-11-thread-1,864802ac622c2c12,d771d84121e626cf] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:03:16.833 INFO 26087 --- [ForkJoinPool-7-worker-2,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:03:16.835 INFO 26087 --- [ForkJoinPool-7-worker-2,864802ac622c2c12,c5f483e0bd6642a3] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=2, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:03:16.835 INFO 26087 --- [pool-11-thread-4,864802ac622c2c12,5bfa94df7f3b9159] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:03:16.838 INFO 26087 --- [pool-11-thread-3,864802ac622c2c12,e18efd6e288401a7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:03:17.021 INFO 26087 --- [ocp-async-7,8ec1a44aeefd3a62,4606659d3df3f135] c.o.o.c.t.e.coordinator.TaskCoordinator : Start task coordinator 2025-04-08 21:03:17.074 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:17.075 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:17.079 INFO 26087 --- [ocp-schedules-24,f66f7c4a76438e39,c668108db225acb5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:03:17.083 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=ocpdb, obClusterId=1744114127, tenantName=sys, username=root, address=192.168.1.61, port=2881, database=oceanbase) 2025-04-08 21:03:17.083 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:03:17.093 WARN 26087 --- [ocp-schedules-24,f66f7c4a76438e39,c668108db225acb5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:03:17.094 INFO 26087 --- [obsdk-init-thread-1,21005228e1ced472,ff0c5d7ffe14cd62] com.alibaba.druid.pool.DruidDataSource : {dataSource-3,ob-connector-datasource_direct_sys_root_192.168.1.61_2881} inited 2025-04-08 21:03:17.094 INFO 26087 --- [obsdk-init-thread-1,21005228e1ced472,ff0c5d7ffe14cd62] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=[ObServerAddr(address=192.168.1.61, port=2881)], username=root, tenantName=sys, clusterName=ocpdb, obClusterId=1744114127, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:03:17.095 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:03:17.096 INFO 26087 --- [ocp-schedules-24,f66f7c4a76438e39,c668108db225acb5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:03:17.098 WARN 26087 --- [ocp-schedules-24,f66f7c4a76438e39,c668108db225acb5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:03:17.105 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:03:17.107 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:17.108 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:03:17.110 INFO 26087 --- [ocp-schedules-24,f66f7c4a76438e39,c668108db225acb5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=1 2025-04-08 21:03:17.123 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:17.134 INFO 26087 --- [ocp-async-7,8ec1a44aeefd3a62,4606659d3df3f135] c.o.o.s.common.config.TaskEngineRunner : Startup subtask coordinator. subtask=623 2025-04-08 21:03:17.134 INFO 26087 --- [ocp-async-7,8ec1a44aeefd3a62,4606659d3df3f135] c.o.o.c.t.e.c.SubtaskCoordinator : Start subtask coordinator 2025-04-08 21:03:17.315 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:03:17.362 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:17.481 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.869028214S 2025-04-08 21:03:17.482 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:03:17.540 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:03:17.540 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:03:17.567 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:03:17.631 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:03:17.631 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:03:17.631 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:03:17.631 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:03:17.631 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:03:17.632 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:03:17.639 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:03:17.644 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:17.744 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] o.s.boot.SpringApplication : Started application in 1.154 seconds (JVM running for 103.117) 2025-04-08 21:03:17.745 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.154735549S 2025-04-08 21:03:17.854 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:03:17.905 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:18.062 INFO 26087 --- [ocp-schedules-17,62ca4cd1f53f1c0b,ed66663d935d5559] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:03:18.093 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select a.tenant_id, count(a.table_id) no_updated_count from (select tenant_id, table_id, table_name from oceanbase.__all_virtual_table tab where table_type in (0, 2, 3, 14) and table_id > 200000 and not exists ( select 1 from oceanbase.__all_virtual_table_stat stat where stat.tenant_id = tab.tenant_id and stat.table_id = tab.table_id ) union all select t.tenant_id, t.table_id, t.table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 0 and tab.table_id > 200000) t, oceanbase.__all_virtual_table_stat stat, (select tenant_id, table_id, cast(sum(inserts + updates + deletes) - sum(last_inserts + last_updates + last_deletes) as signed) as inc_mod_count from oceanbase.__all_virtual_monitor_modified group by tenant_id, table_id ) monitor where t.tenant_id = stat.tenant_id and t.table_id = stat.table_id and stat.last_analyzed <= date_sub(now(), INTERVAL ? DAY) and t.tenant_id = monitor.tenant_id and t.table_id = monitor.table_id and monitor.inc_mod_count / stat.row_cnt * 100 > t.stale_percent union all select t.tenant_id, t.table_id, any_value(t.table_name) table_name from (select tab.tenant_id, tab.table_id, tab.table_name, cast(ifnull(u_p.valchar, g_p.spare4) as signed) stale_percent from oceanbase.__all_virtual_table tab join oceanbase.__all_virtual_optstat_global_prefs g_p on g_p.tenant_id = tab.tenant_id and g_p.sname = 'STALE_PERCENT' left join oceanbase.__all_virtual_optstat_user_prefs u_p on u_p.tenant_id = tab.tenant_id and u_p.table_id = tab.table_id and u_p.pname = 'STALE_PERCENT' where tab.table_type in (0, 2, 3, 14) and tab.part_level = 1 and tab.table_id > 2025-04-08 21:03:18.138 INFO 26087 --- [ocp-async-3,21005228e1ced472,80c6eeb0ab0813ed] o.o.s.g.s.StatsGatherOverviewDetailCache : Sync getNoUpdatedStatsGatherStatisticsMap ok, statisticMap = {} 2025-04-08 21:03:18.355 INFO 26087 --- [prometheus-minute-schedule-1,7928a0e0e66e09f0,d3e9ef595c4e4340] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:18.535 WARN 26087 --- [ocp-async-10,d634defdf634a97d,c4e408e152735a2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:18.535 INFO 26087 --- [ocp-async-1,a60fb0b9c4527c37,3c28fdf46480dede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:20.075 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:20.076 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:20.535 INFO 26087 --- [ocp-async-7,cfb208813a27c643,9dcf8d74b870179d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:20.535 WARN 26087 --- [ocp-async-1,b8584d5241cf7d09,ede6f36bff748937] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:03:20.627 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2025-04-08 21:03:20.627 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2025-04-08 21:03:20.632 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] o.s.web.servlet.DispatcherServlet : Completed initialization in 5 ms 2025-04-08 21:03:20.808 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,983a1d66c33488b4,ae6da227581952f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=983a1d66c33488b4, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.811 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,983a1d66c33488b4,ae6da227581952f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=983a1d66c33488b4, duration=147 ms] 2025-04-08 21:03:20.882 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,795653c96eb2fd38,b8bd1be09380fe48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=795653c96eb2fd38, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.882 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,795653c96eb2fd38,b8bd1be09380fe48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=795653c96eb2fd38, duration=3 ms] 2025-04-08 21:03:20.890 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,78d5a806812bdd38,2f58dd6e114e4cfb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=78d5a806812bdd38, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.890 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,78d5a806812bdd38,2f58dd6e114e4cfb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=78d5a806812bdd38, duration=2 ms] 2025-04-08 21:03:20.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0d0c4d77f36f5873,d677510e3efbdf7f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=0d0c4d77f36f5873, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0d0c4d77f36f5873,d677510e3efbdf7f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=0d0c4d77f36f5873, duration=2 ms] 2025-04-08 21:03:20.904 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,19e109555f8ff83d,86d33a80147d8e6e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=19e109555f8ff83d, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.904 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,19e109555f8ff83d,86d33a80147d8e6e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=19e109555f8ff83d, duration=2 ms] 2025-04-08 21:03:20.911 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,947efed958b9b738,0a3a71b331d20f2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=947efed958b9b738, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.911 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,947efed958b9b738,0a3a71b331d20f2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=947efed958b9b738, duration=2 ms] 2025-04-08 21:03:20.917 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,0fb36974067b2766,3e7019df90264ca4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=0fb36974067b2766, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.917 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,0fb36974067b2766,3e7019df90264ca4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=0fb36974067b2766, duration=2 ms] 2025-04-08 21:03:20.923 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,d74f9c9256051bde,a44e93884635baab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=d74f9c9256051bde, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.923 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,d74f9c9256051bde,a44e93884635baab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=d74f9c9256051bde, duration=1 ms] 2025-04-08 21:03:20.931 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,1dafeaa6db2d8c85,58f9b065412d8dd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=1dafeaa6db2d8c85, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.931 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,1dafeaa6db2d8c85,58f9b065412d8dd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=1dafeaa6db2d8c85, duration=2 ms] 2025-04-08 21:03:20.938 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,8257611da9fb2f05,011e661474750d33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=8257611da9fb2f05, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.938 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,8257611da9fb2f05,011e661474750d33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=8257611da9fb2f05, duration=2 ms] 2025-04-08 21:03:20.998 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,969b3bd624ac69c8,55c4d2aedd993bf1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=192.168.1.61, traceId=969b3bd624ac69c8, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:03:20.998 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,969b3bd624ac69c8,55c4d2aedd993bf1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=192.168.1.61, traceId=969b3bd624ac69c8, duration=2 ms] 2025-04-08 21:03:21.355 INFO 26087 --- [prometheus-minute-schedule-1,878813c9eb7f440a,2df6aafd0fdcde01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:22.536 WARN 26087 --- [ocp-async-9,22782f2f31cc7037,ecd836a3a2c12cfc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:22.536 INFO 26087 --- [ocp-async-5,09187d1088eb9dbd,5c3259f318bfd188] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:23.076 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:23.077 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:24.355 INFO 26087 --- [prometheus-minute-schedule-1,44919aa7e2659d71,6c3046f7586698b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:24.536 WARN 26087 --- [ocp-async-5,f06e8dcea03a0fb4,0f7f7a4177a0af40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:24.536 INFO 26087 --- [ocp-async-10,0eb1337dcd1e48e1,9999eb48a1be3c6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:26.077 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:26.079 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:26.535 INFO 26087 --- [ocp-async-1,50b2f87aa2271d6b,ac99f757de22f246] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:03:26.535 INFO 26087 --- [ocp-schedules-16,f307578dd2116295,f88cf8fa10a3802f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:03:26.536 WARN 26087 --- [ocp-async-3,c608bc8ae6dc6720,27e678e66186f45b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:26.536 INFO 26087 --- [ocp-async-9,8a20ce0f8ae824e2,10f89e8fa2e8dd10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:26.548 INFO 26087 --- [ocp-schedules-23,277cea69af3f9bb0,4ed4d404283c0259] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 21:03:26.552 INFO 26087 --- [ocp-schedules-8,5fd617caa6531a02,06c5eb6109d6c1a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:03:26.553 INFO 26087 --- [ocp-schedules-10,8165bcc95347a9a0,1de5be302e2ce085] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:03:26.570 INFO 26087 --- [ocp-schedules-7,d342b5c38374101c,983157b16c354499] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:03:26.635 INFO 26087 --- [ocp-schedules-14,c4d196e37040da10,7f399cab3ed90d1e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:03:26.636 INFO 26087 --- [ocp-schedules-14,c4d196e37040da10,7f399cab3ed90d1e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:03:27.042 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:45:00.0, nextRun=2025-04-08 20:50:00.0, currentDate=Tue Apr 08 21:03:27 CST 2025 2025-04-08 21:03:27.042 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:45:00.0, nextRunTime=2025-04-08 20:50:00.0} 2025-04-08 21:03:27.053 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:03:27.055 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:03:27.054+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:03:27.210 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Start cache subtask classes 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 21:03:27.211 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRootInspectTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSyncClusterInfoTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeClusterHealthCheckerScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandbySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$RemoteStandbySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemoteStandbySenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalPrimary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$LocalPrimary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalPrimary 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McPrepareUpgradeClusterTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemoteStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$RemoteStandby 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalStandby, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$LocalStandby, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$LocalStandby 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePreScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemoteSenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$RemoteSenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecoverObParameterTask$RemoteSenderTask 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeCheckerScriptTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McCleanUpgradeFilesTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McEndUpgradeTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Primary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$Primary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McSetClusterContextTask$Primary 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Remote, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McBeginUpgradeTask$Remote, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Remote 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=RemotePrimarySenderTask, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPreScriptTask$RemotePrimarySenderTask, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$RemotePrimarySenderTask 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=LocalPrimary, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradeRollingPostScriptTask$LocalPrimary, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McExecUpgradePostCheckerScriptTask$LocalPrimary 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=Local, class=com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McUpdateObClusterStatusTask$Local, existClass=class com.oceanbase.ocp.service.task.business.mc.operation.ob.upgrade.McRecordObParameterTask$Local 2025-04-08 21:03:27.213 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=UpdateObClusterStatusTask, class=com.oceanbase.ocp.obops.internal.cluster.task.UpdateObClusterStatusTask, existClass=class com.oceanbase.ocp.service.task.business.host.UpdateObClusterStatusTask 2025-04-08 21:03:27.214 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask simple name already exist, remove exist info. SimpleName=CreateResourcePoolTask, class=com.oceanbase.ocp.backup.internal.task.schedule.CreateResourcePoolTask, existClass=class com.oceanbase.ocp.service.task.business.tenant.CreateResourcePoolTask 2025-04-08 21:03:27.214 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Remove duplicated simple name cache, size=47 2025-04-08 21:03:27.214 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SubtaskClassCache : Subtask cache initialized, simpleNameSize=565, classNameSize=623 2025-04-08 21:03:27.229 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:45:00.0, nextRun=2025-04-08 20:50:00.0, currentDate=Tue Apr 08 21:03:27 CST 2025 2025-04-08 21:03:27.229 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 20:45:00.0, nextRunTime=2025-04-08 20:50:00.0} 2025-04-08 21:03:27.238 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:03:27.288 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:03:27.288 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:03:27.298 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2025-04-08 21:01:00.0, currentDate=Tue Apr 08 21:03:27 CST 2025 2025-04-08 21:03:27.299 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=null, nextRunTime=2025-04-08 21:01:00.0} 2025-04-08 21:03:27.302 WARN 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2025-04-08 21:03:27.302 WARN 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2025-04-08 21:03:27.302 WARN 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:373) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:178) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:139) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:61) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:138) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:107) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:136) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:74) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:03:27.306 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2025-04-08 21:03:27.309 WARN 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=1000002 2025-04-08 21:03:27.317 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 20:40:05.0, nextRun=2025-04-08 20:50:04.0, currentDate=Tue Apr 08 21:03:27 CST 2025 2025-04-08 21:03:27.317 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-04-08 20:40:05.0, nextRunTime=2025-04-08 20:50:04.0} 2025-04-08 21:03:27.322 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 21:03:27.322 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:03:27.322+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:03:27.332 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=null, nextRun=2025-04-08 21:00:00.0, currentDate=Tue Apr 08 21:03:27 CST 2025 2025-04-08 21:03:27.332 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=1000001, name=Collect backup files size on schedule, templateName=BackupScheduleCollectTemplateFactory, scheduleRule=0 0 */3 * * *, lastRunTime=null, nextRunTime=2025-04-08 21:00:00.0} 2025-04-08 21:03:27.339 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Collect backup files size, nodeCount=1, taskCount=1 2025-04-08 21:03:27.339 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,f09ea1f9584a6170] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Collect backup files size', taskDefinitionId=1000001, state=RUNNING, type=SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:03:27.339+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Collect backup files size, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.backup.internal.task.BackupCollectSizeTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:03:27.355 INFO 26087 --- [prometheus-minute-schedule-1,2614fda77cc41f47,264235a2728dbafd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:28.179 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,6d8d968efbee9e4a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000001, operation:EXECUTE, state: RUNNING 2025-04-08 21:03:28.207 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,6d8d968efbee9e4a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000002, operation:EXECUTE, state: RUNNING 2025-04-08 21:03:28.224 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,6d8d968efbee9e4a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000003, operation:EXECUTE, state: RUNNING 2025-04-08 21:03:28.233 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:03:28.236 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000001, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000001, task_operation=execute, latest_execution_start_time=2025-04-08T21:03:28.167+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000001}, listMap={}}, executor=192.168.1.61 2025-04-08 21:03:28.243 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.249 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:03:28.252 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000002, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000003, task_operation=execute, latest_execution_start_time=2025-04-08T21:03:28.204+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=1000002}, listMap={}}, executor=192.168.1.61 2025-04-08 21:03:28.257 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:03:28.258 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:03:28.259 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_monitor, username=root, address=192.168.1.61, port=2881, database=monitor_database) 2025-04-08 21:03:28.259 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2025-04-08 21:03:28.262 INFO 26087 --- [obsdk-init-thread-2,8ec1a44aeefd3a62,73b10885a26f43c9] com.alibaba.druid.pool.DruidDataSource : {dataSource-4,ob-connector-datasource_direct_ocp_monitor_root_192.168.1.61_2881} inited 2025-04-08 21:03:28.262 INFO 26087 --- [obsdk-init-thread-2,8ec1a44aeefd3a62,73b10885a26f43c9] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_monitor, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=monitor_database) 2025-04-08 21:03:28.262 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 21:03:28.262 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:03:28.263 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:03:28.264 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.267 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.270 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.271 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:03:28.271 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.272 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.273 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.274 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000003, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000004, task_operation=execute, latest_execution_start_time=2025-04-08T21:03:28.222+08:00, sub_task_instance_name=Collect backup files size, sub_task_instance_id=1000003}, listMap={}}, executor=192.168.1.61 2025-04-08 21:03:28.274 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.277 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:03:28.280 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.280 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.282 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.b.i.m.ProxyBackupStorageMonitor : collect backup storage info for clusters: 2025-04-08 21:03:28.282 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.317 INFO 26087 --- [subtask-executor1,8ec1a44aeefd3a62,12208efaf2b929d7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000001, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:03:28.324 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.b.i.m.ProxyBackupStorageMonitor : collectInfoList=[] 2025-04-08 21:03:28.325 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:03:28.331 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.b.i.m.s.BackupCollectInfoProvider : storageCollectInfoList size:[0] 2025-04-08 21:03:28.346 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.b.i.m.s.StorageCapacityExporter : Start pre hook computers 2025-04-08 21:03:28.350 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.b.i.m.s.StorageCapacityExporter : Start post hook computers 2025-04-08 21:03:28.355 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.356 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.357 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.357 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:03:28.360 INFO 26087 --- [subtask-executor3,8ec1a44aeefd3a62,89b8ba83a6edd896] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000003, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:03:28.368 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.368 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.369 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.369 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:03:28.382 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.382 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.383 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.383 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:03:28.395 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.396 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.396 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.397 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:03:28.402 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.402 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.403 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.403 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:03:28.408 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.409 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.410 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.410 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:03:28.414 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.415 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.415 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.415 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:03:28.420 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.421 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.421 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.421 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:03:28.426 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.426 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.427 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.427 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:03:28.431 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.432 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.432 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.432 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:03:28.440 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.440 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.441 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.441 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:03:28.452 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.453 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.453 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.453 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:03:28.458 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.458 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.459 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.459 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:03:28.464 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.464 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.465 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.465 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:03:28.470 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.471 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.471 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.472 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:03:28.477 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.477 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.478 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.479 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:03:28.482 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.483 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.483 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.483 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:03:28.487 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.487 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.488 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.488 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:03:28.492 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.492 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.493 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.493 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:03:28.496 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.496 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.497 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.497 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:03:28.501 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.501 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.502 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.502 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:03:28.506 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.506 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.506 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.507 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:03:28.510 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.511 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.511 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.511 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:03:28.515 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.515 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.516 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.516 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:03:28.520 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.520 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.520 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.521 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:03:28.525 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.525 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.525 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.526 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:03:28.529 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.530 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.530 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.530 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:03:28.534 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.535 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.535 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.536 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:03:28.536 WARN 26087 --- [ocp-async-7,bff7a7d39b63dbc0,ebbae900c67c2562] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:28.536 INFO 26087 --- [ocp-async-5,ca3afdd867f73012,055523359da7ae74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:28.541 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.541 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.541 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.542 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:03:28.545 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.545 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.546 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.546 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:03:28.549 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.550 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.550 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.550 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:03:28.554 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.554 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.555 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.555 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:03:28.558 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.559 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.559 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.559 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:03:28.566 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.566 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.567 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.567 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:03:28.573 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.573 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.574 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.574 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:03:28.579 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.579 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.579 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.580 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:03:28.584 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.584 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.584 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.585 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:03:28.588 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.588 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.589 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.589 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:03:28.593 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.593 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.594 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.594 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:03:28.598 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.598 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.599 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.599 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:03:28.602 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.602 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.604 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.604 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:03:28.612 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.612 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.613 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.613 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:03:28.630 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.630 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.631 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.631 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:03:28.636 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.636 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.637 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.637 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:03:28.642 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.642 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.643 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.643 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:03:28.648 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.648 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.648 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.649 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:03:28.652 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.652 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.653 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.653 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:03:28.656 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.657 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.657 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.657 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:03:28.661 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:03:28.661 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:03:28.661 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=ocp_meta, username=root, address=192.168.1.61, port=2881, database=meta_database) 2025-04-08 21:03:28.661 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_meta, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2025-04-08 21:03:28.664 INFO 26087 --- [obsdk-init-thread-3,8ec1a44aeefd3a62,7f6b74200f66f4dd] com.alibaba.druid.pool.DruidDataSource : {dataSource-5,ob-connector-datasource_direct_ocp_meta_root_192.168.1.61_2881} inited 2025-04-08 21:03:28.664 INFO 26087 --- [obsdk-init-thread-3,8ec1a44aeefd3a62,7f6b74200f66f4dd] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=ocp_meta, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=meta_database) 2025-04-08 21:03:28.664 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:03:28.666 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.666 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:03:28.668 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.668 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.669 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.670 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:03:28.674 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.674 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:28.675 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:28.675 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:03:28.678 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:03:28.696 INFO 26087 --- [subtask-executor2,8ec1a44aeefd3a62,cbe5231acda657f1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000002, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:03:29.079 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:29.079 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:29.100 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000001, status:SUCCESS 2025-04-08 21:03:29.106 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000001, status:SUCCESS 2025-04-08 21:03:29.119 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000003, status:SUCCESS 2025-04-08 21:03:29.121 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000003, status:SUCCESS 2025-04-08 21:03:29.127 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000004, status:SUCCESS 2025-04-08 21:03:29.127 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000004, status:SUCCESS 2025-04-08 21:03:30.355 INFO 26087 --- [prometheus-minute-schedule-1,d6aae73ea50565b4,95aa764be8b149f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:30.537 WARN 26087 --- [ocp-async-4,5e1ce34e0480e3c9,ae80bfc36a433cd9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:30.537 INFO 26087 --- [ocp-async-3,01fa5edcfa0bae74,418098ca72398be3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:31.751 INFO 26087 --- [part-initial-worker,,] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:03:31.751 INFO 26087 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:03:31.751 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.752 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:03:31.753 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.753 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.753 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.754 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:03:31.766 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.766 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.767 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.767 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:03:31.778 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.778 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.778 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.779 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:03:31.792 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.792 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.793 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.793 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:03:31.805 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.806 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.806 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.806 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:03:31.811 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.812 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.812 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.812 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:03:31.816 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.816 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.817 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.817 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:03:31.822 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.823 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.823 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.824 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:03:31.829 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.829 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.830 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.830 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:03:31.835 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.836 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.836 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.836 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:03:31.841 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.841 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.841 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.841 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:03:31.848 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.848 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.848 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.849 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:03:31.860 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.860 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.860 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.861 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:03:31.865 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.865 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.865 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.866 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:03:31.870 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.871 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.871 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.871 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:03:31.875 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.876 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.876 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.876 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:03:31.880 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.880 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.881 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.881 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:03:31.884 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.884 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.885 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.885 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:03:31.889 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.889 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.889 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.890 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:03:31.893 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.894 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.894 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.894 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:03:31.897 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.897 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.898 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.898 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:03:31.901 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.902 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.902 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.902 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:03:31.906 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.906 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.906 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.907 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:03:31.911 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.911 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.911 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.911 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:03:31.915 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.915 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.915 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.915 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:03:31.919 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.919 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.919 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.920 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:03:31.923 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.923 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.924 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.924 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:03:31.928 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.928 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.929 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.929 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:03:31.932 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.932 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.933 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.933 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:03:31.937 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.937 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.938 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.938 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:03:31.941 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.942 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.942 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.942 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:03:31.946 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.946 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.947 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.947 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:03:31.950 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.951 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.951 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.951 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:03:31.955 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.955 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.955 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.956 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:03:31.961 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.961 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.962 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.962 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:03:31.966 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.967 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.967 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.967 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:03:31.972 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.972 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.972 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.973 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:03:31.977 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.977 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.978 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.978 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:03:31.981 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.981 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.981 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.982 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:03:31.984 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.985 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.985 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.985 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:03:31.988 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.988 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.989 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.989 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:03:31.992 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.992 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:31.993 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:31.993 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:03:32.001 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.001 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.001 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.002 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:03:32.018 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.019 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.019 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.019 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:03:32.024 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.025 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.025 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.025 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:03:32.030 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.030 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.031 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.031 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:03:32.036 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.036 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.036 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.037 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:03:32.040 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.040 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.040 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.040 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:03:32.044 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.045 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.045 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.045 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:03:32.049 INFO 26087 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:03:32.049 INFO 26087 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:03:32.050 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.050 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:03:32.051 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.051 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.052 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.052 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:03:32.056 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.056 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:03:32.057 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:32.057 INFO 26087 --- [part-initial-worker,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:03:32.060 INFO 26087 --- [part-initial-worker,,] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:03:32.079 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:32.080 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:32.537 WARN 26087 --- [ocp-async-2,fb620a2b86e9ffed,b03a5db38781a502] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:32.537 INFO 26087 --- [ocp-async-8,3dc54f5edb789753,f9fe27bf5c6366c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:33.355 INFO 26087 --- [prometheus-minute-schedule-1,c86ed269f66e970c,a81687023f44bd7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:34.537 WARN 26087 --- [ocp-async-7,583a503ece302c9d,815c90cd74fb4393] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:34.537 INFO 26087 --- [ocp-async-1,f14585be13136a70,6667673707dbfb26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:34.979 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:34.979 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:03:34.980 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:03:34.980 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:03:34.981 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:03:34.987 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(2), status=TAKINGOVER 2025-04-08 21:03:34.988 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=6ms, period=120000ms 2025-04-08 21:03:35.080 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:35.081 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:36.355 INFO 26087 --- [prometheus-minute-schedule-1,682991b02632d8f7,eff1ce028905b722] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:36.535 INFO 26087 --- [ocp-async-1,11a69d0585c28a77,15f092bedc4b0b53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:03:36.537 INFO 26087 --- [ocp-schedules-23,a69bd863da0ba312,4438057a3f34596d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:03:36.537 WARN 26087 --- [ocp-async-4,c97b69d9d7dd16f9,7c49cad4f9806117] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:36.537 INFO 26087 --- [ocp-async-10,d03e7d580a9f948f,a3beaa6c07eae0eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:36.642 INFO 26087 --- [ocp-schedules-13,f8f04facc61f19a3,916a19fcb12b3cf5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:03:36.644 INFO 26087 --- [ocp-schedules-13,f8f04facc61f19a3,916a19fcb12b3cf5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:03:38.081 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:38.082 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:38.537 WARN 26087 --- [ocp-async-8,5542f5d9c3eb7342,516f7e1818da28cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:38.538 INFO 26087 --- [ocp-async-2,b6f6c6b6a2cfba1f,fada27b3cb343d91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:39.355 INFO 26087 --- [prometheus-minute-schedule-1,d8e861e3240f3842,4450cf8817ec5560] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:40.003 INFO 26087 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:03:40.003 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:03:40.538 WARN 26087 --- [ocp-async-5,07523887032c8b7c,5f1d41447ebbc1ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:40.538 INFO 26087 --- [ocp-async-7,9d7d2364646ee78e,beb96e8a7e12e675] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:41.082 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:41.083 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:42.355 INFO 26087 --- [prometheus-minute-schedule-1,db15d1b99a0a8e0b,e12c09e84e00a274] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:42.538 WARN 26087 --- [ocp-async-4,4484f18b395cb6db,a1819121705b934c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:42.538 INFO 26087 --- [ocp-async-6,008f2865c59af206,d0eee87843617fa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:44.083 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:44.084 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:44.538 WARN 26087 --- [ocp-async-3,5bc8895ab044d7b7,ca1e6f735ceae4ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:44.538 INFO 26087 --- [ocp-async-10,469f147879597470,46290f44c587d8bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:45.355 INFO 26087 --- [prometheus-minute-schedule-1,444f5b841482a4a8,bdb0795712351f32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:46.521 INFO 26087 --- [ocp-schedules-4,9fbb835ae81743ab,b1a12611d63a0c7e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:03:46.524 INFO 26087 --- [ocp-schedules-24,c9a86221679ac78e,b4a878a8612a79bc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:03:46.524 INFO 26087 --- [ocp-schedules-9,aff7f5bbf32f2f94,898fa33389c2ce20] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:03:46.524 INFO 26087 --- [ocp-schedules-8,cf6db57a58905f6f,a361cfaaa67c45a0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:03:46.525 INFO 26087 --- [ocp-schedules-24,c9a86221679ac78e,b4a878a8612a79bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:46.526 INFO 26087 --- [ocp-schedules-9,aff7f5bbf32f2f94,898fa33389c2ce20] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:46.526 INFO 26087 --- [ocp-schedules-8,cf6db57a58905f6f,a361cfaaa67c45a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:03:46.526 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:03:46.532 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 21:03:46.532 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:03:46.533 INFO 26087 --- [Sync all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_sync', labels={app=OB, failed_reason=-, ob_cluster_id=1744114127, ob_cluster_name=ocpdb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744117426 : 2025-04-08 13:03:46 2025-04-08 21:03:46.535 INFO 26087 --- [ocp-async-5,e848cd7f7cab41a4,b94fec8541ff120d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:03:46.539 INFO 26087 --- [ocp-schedules-20,598a2dddc7d4577e,b841492aa22f4c83] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:03:46.539 WARN 26087 --- [ocp-async-9,baaba71330f3e627,490877584d7492c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:46.539 INFO 26087 --- [ocp-async-7,e2dd6af8eda06a24,8f018d945ca2349d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:46.540 INFO 26087 --- [ocp-schedules-14,fc280aa81ddb896e,dcb32c70ac543da6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:03:46.569 INFO 26087 --- [ocp-schedules-19,549ed864fe0b23ea,2ae19430e5d44b15] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:03:46.655 INFO 26087 --- [ocp-schedules-12,3bf65ec7fe33627d,7cea6e96cf33dba2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:03:46.657 INFO 26087 --- [ocp-schedules-12,3bf65ec7fe33627d,7cea6e96cf33dba2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:03:47.085 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:47.085 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:48.355 INFO 26087 --- [prometheus-minute-schedule-1,44549050bfff3035,71c2dbe26f406b5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:48.539 WARN 26087 --- [ocp-async-6,6ec191d431a5a66f,f5bb2674056ac4d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:48.539 INFO 26087 --- [ocp-async-4,c45b809cb41ba9a0,215760fc534232fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:50.085 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:50.086 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:50.539 WARN 26087 --- [ocp-async-3,91516af91991d1ee,7e6b40236104938b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:50.539 INFO 26087 --- [ocp-async-8,fb1a95a0d0ff0903,38133ffb67128659] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:51.355 INFO 26087 --- [prometheus-minute-schedule-1,e97aa313d2184e65,16c24bc9768ac31b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:52.540 WARN 26087 --- [ocp-async-5,5d4a580ef4099767,37d903e2c4c30b9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:52.540 INFO 26087 --- [ocp-async-9,181e408512194ef3,e6d1fa08dc3acc60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:53.086 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:53.088 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:54.355 INFO 26087 --- [prometheus-minute-schedule-1,d9419d533bf74b87,2f0378bbe748aec1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:54.540 WARN 26087 --- [ocp-async-1,aa76b8686a41ee9a,e2773e73056be77d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:54.540 INFO 26087 --- [ocp-async-6,c35c4c91eabda0e3,54606db0d7a7f119] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:56.088 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:56.089 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:03:56.527 INFO 26087 --- [ocp-schedules-8,cf6db57a58905f6f,a361cfaaa67c45a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:03:56.527 INFO 26087 --- [ocp-schedules-9,aff7f5bbf32f2f94,898fa33389c2ce20] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:03:56.535 INFO 26087 --- [ocp-async-6,5dba58a2dfa647c6,c370da82c6aea946] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:03:56.540 WARN 26087 --- [ocp-async-3,18753c86c44b2750,ce475ef202bb63e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:56.540 INFO 26087 --- [ocp-schedules-1,e1d019267674bd67,2e7fd23ac8b26e4f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:03:56.540 INFO 26087 --- [ocp-async-4,1adcac7a389a6175,98464a678dfd442d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:56.694 INFO 26087 --- [ocp-schedules-26,d1a1618b8da2d725,1d323376359e8d23] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:03:56.696 INFO 26087 --- [ocp-schedules-26,d1a1618b8da2d725,1d323376359e8d23] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:03:57.355 INFO 26087 --- [prometheus-minute-schedule-1,43a6314fcdd440ec,3bb652445b4ceb63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117440, interval=60 2025-04-08 21:03:58.540 WARN 26087 --- [ocp-async-5,aab062fd438dfc7d,de9e7d374a07a7a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:03:58.540 INFO 26087 --- [ocp-async-4,7269a33fe3995e8d,14ec1f524533c5c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:03:59.089 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:03:59.090 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:00.355 INFO 26087 --- [prometheus-minute-schedule-1,9083e52985c9a471,0813d2a6b3175154] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:00.540 WARN 26087 --- [ocp-async-1,7e4b8ccb0a3d95e5,c87fa0476743a5b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:00.540 INFO 26087 --- [ocp-async-4,08da289f977702a7,8e3119ca485e942d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:02.090 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:02.091 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:02.540 WARN 26087 --- [ocp-async-2,21723c839633c863,c949c8b57c429f0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:02.541 INFO 26087 --- [ocp-async-4,0851d11384e337da,082304e4b98ef557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:03.355 INFO 26087 --- [prometheus-minute-schedule-1,c08a91059e70ec91,34ce0979f75f43ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:04.541 WARN 26087 --- [ocp-async-10,56b2db2f7c8cf56a,4f48ae83b281519d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:04.541 INFO 26087 --- [ocp-async-5,7c16d1359cb9aee0,0714dcc0ffa62960] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:05.091 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:05.092 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:06.167 WARN 26087 --- [pool-18-thread-1,,] o.p.s.d.i.OcpPerfSqlDiagnosisTaskService : OcpPerfSqlDiagnosisTaskEntity(taskName=PLAN_CHANGE_DETECTOR, beginIntervalTime=1744116193253000, endIntervalTime=1744116313266000, executor=192.168.1.61, status=SUCCESS, createTime=2025-04-08T12:47:13.265Z, updateTime=2025-04-08T12:47:13.272Z) is too old, skip to diag OcpPerfSqlDiagnosisTaskEntity(taskName=PLAN_CHANGE_DETECTOR, beginIntervalTime=1744116726167000, endIntervalTime=1744117326167000, executor=192.168.1.61, status=RUNNING, createTime=null, updateTime=null) 2025-04-08 21:04:06.170 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T20:44:12.515, end=2025-04-08T21:03:06.157} 2025-04-08 21:04:06.180 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:04:06.181 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T20:44:12.515, end=2025-04-08T21:03:06.167} 2025-04-08 21:04:06.181 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T20:52:06.167, end=2025-04-08T21:02:06.167} 2025-04-08 21:04:06.182 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:04:06.185 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:04:06.185 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:04:06.185 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:04:06.185 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:04:06.355 INFO 26087 --- [prometheus-minute-schedule-1,5465efd54808abfd,a2b5e15447770c14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:06.355 INFO 26087 --- [prometheus-cache-monitor1,a7c931d2dbf4a376,90db9eb255d62de0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 21:04:06.356 INFO 26087 --- [prometheus-cache-monitor1,a7c931d2dbf4a376,90db9eb255d62de0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:04:06.536 INFO 26087 --- [ocp-async-9,dc60fcd46211100b,6437afeba6070f93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:04:06.541 WARN 26087 --- [ocp-async-3,a3bded9edd5bdbb3,4df2e8db0351c61a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:06.541 INFO 26087 --- [ocp-async-7,fa6b7e9cfd313db3,a048e87d2c389720] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:06.542 INFO 26087 --- [ocp-schedules-15,61ec4fe941fb58fc,5c7329ec8527077d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:04:06.700 INFO 26087 --- [ocp-schedules-20,1afa1c28a436a272,80d90ba6e6298c09] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:04:06.702 INFO 26087 --- [ocp-schedules-20,1afa1c28a436a272,80d90ba6e6298c09] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:04:08.092 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:08.093 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:08.541 WARN 26087 --- [ocp-async-6,07aef5fe81130cab,ebb78323b6b23c66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:08.541 INFO 26087 --- [ocp-async-2,1b4e2189a19b6c59,e4a8146925c76d6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:09.355 INFO 26087 --- [prometheus-minute-schedule-1,c047f3b5b270f573,4ffcd813d07b1209] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:10.002 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:04:10.002 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:04:10.541 WARN 26087 --- [ocp-async-2,152731a43181fd9f,1eb5b2847fcc2966] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:10.541 INFO 26087 --- [ocp-async-4,94407097f6773ee2,7aedca034add2ff6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:10.633 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:04:11.093 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:11.094 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:12.355 INFO 26087 --- [prometheus-minute-schedule-1,979917b0b305cf9a,7800b47887412b9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:12.541 WARN 26087 --- [ocp-async-3,05ef26d6cd2e489e,5c9941ad3169917d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:12.541 INFO 26087 --- [ocp-async-1,5085b05a50c1538c,c3383550599b4803] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:14.094 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:14.095 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:14.542 WARN 26087 --- [ocp-async-1,5ad83cb424690afa,20f1f17c8f3091ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:14.542 INFO 26087 --- [ocp-async-6,d369d81c7a442ff5,8f973f31018b6b78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:15.355 INFO 26087 --- [prometheus-minute-schedule-1,ba9dd8fe6e56b5ab,2213bc801a172f12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:16.517 INFO 26087 --- [ocp-schedules-22,6a55d761f4b44536,07393f5383b98445] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:04:16.518 INFO 26087 --- [ocp-schedules-7,0ac745159d5d3db7,2a64a5ad5c569e92] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:04:16.521 INFO 26087 --- [ocp-schedules-14,550cfc307c1f1cb5,a1f0e6592654b6bc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:04:16.524 INFO 26087 --- [ocp-schedules-7,0ac745159d5d3db7,2a64a5ad5c569e92] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:04:16.524 INFO 26087 --- [ocp-schedules-17,a322ef5b1fac82aa,0b053d7bf6cf5eb4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:04:16.524 INFO 26087 --- [ocp-schedules-23,c956c0e6fad5c907,78e310d2a7e5be46] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:04:16.525 INFO 26087 --- [ocp-schedules-23,c956c0e6fad5c907,78e310d2a7e5be46] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:16.525 INFO 26087 --- [ocp-schedules-17,a322ef5b1fac82aa,0b053d7bf6cf5eb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:04:16.528 INFO 26087 --- [ocp-schedules-22,6a55d761f4b44536,07393f5383b98445] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:04:16.533 INFO 26087 --- [ocp-schedules-27,6a3a2269881a34c6,5b993471d5004997] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:04:16.533 INFO 26087 --- [ocp-schedules-24,c9a86221679ac78e,b4a878a8612a79bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:04:16.533 INFO 26087 --- [ocp-schedules-24,92d4473806b52045,c46001a698e6dfff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:04:16.534 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.ocp.alarm.core.task.AlarmDetectTask : refresh host alarm wrapper 2025-04-08 21:04:16.534 INFO 26087 --- [ocp-schedules-24,92d4473806b52045,c46001a698e6dfff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:16.539 INFO 26087 --- [ocp-schedules-30,fa7630f00fcc082f,6904f4c6376e201e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:04:16.540 INFO 26087 --- [ocp-async-5,81b7b659587f4de6,bbd124017ba669a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:04:16.547 INFO 26087 --- [ocp-schedules-10,0925785f29205545,c581370e5e684804] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:04:16.547 INFO 26087 --- [ocp-async-4,31553f6269d79a21,93db73e949291943] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:16.547 WARN 26087 --- [ocp-async-9,d5608b19b5f87ff0,704449d1af6c02e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:16.551 INFO 26087 --- [ocp-schedules-30,fa7630f00fcc082f,6904f4c6376e201e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:04:16.556 INFO 26087 --- [ocp-schedules-20,df51ad777be253cd,0a38d3a022d4e1d2] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 21:04:16.556 INFO 26087 --- [ocp-schedules-20,df51ad777be253cd,0a38d3a022d4e1d2] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 21:04:16.564 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.ocp.alarm.core.task.AlarmDetectTask : refresh alarm target 2025-04-08 21:04:16.564 INFO 26087 --- [ocp-schedules-12,f5561ba9a1137b99,476a133023e951d8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:04:16.565 INFO 26087 --- [ocp-schedules-6,3fb0ae7e650ae0c8,2967b54157ba67cf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:04:16.565 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 21:04:16.565 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:04:16.566 INFO 26087 --- [ocp-schedules-12,f5561ba9a1137b99,476a133023e951d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:16.566 INFO 26087 --- [ocp-schedules-6,3fb0ae7e650ae0c8,2967b54157ba67cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:16.568 INFO 26087 --- [ocp-schedules-3,bad114704702c4a4,86ab0378b44458b4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:04:16.568+08:00, clusters=[2] 2025-04-08 21:04:16.572 INFO 26087 --- [ocp-schedules-9,802959983ce65bed,335c51961b39e2f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:04:16.573 INFO 26087 --- [ocp-schedules-3,bad114704702c4a4,86ab0378b44458b4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:04:16.573 INFO 26087 --- [ocp-schedules-1,dce0912da8c7e067,f8bc6d11da513f79] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:04:16.574 INFO 26087 --- [ocp-schedules-28,059630e725c36e49,76b80e483a61a301] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:04:16.575 INFO 26087 --- [ocp-schedules-7,7d1308d4c355d0c3,e677b6b7de6f4708] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:04:16.575+08:00, clusters=[2] 2025-04-08 21:04:16.576 INFO 26087 --- [ocp-schedules-7,7d1308d4c355d0c3,e677b6b7de6f4708] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:04:16.585 INFO 26087 --- [ocp-schedules-19,75608775d3cd0962,62916df3f55f3ba5] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:04:16.585 INFO 26087 --- [ocp-schedules-19,75608775d3cd0962,62916df3f55f3ba5] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:04:16.589 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:04:16.589 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:04:16.592 INFO 26087 --- [ocp-schedules-15,adf6dd2f66a0b5bb,883b54d3b4b80ee1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:04:16.594 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=1 2025-04-08 21:04:16.594 INFO 26087 --- [ocp-schedules-11,005d5de80f48d050,2fbd17eaaa1f3d95] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:04:16.594 INFO 26087 --- [ocp-schedules-11,005d5de80f48d050,2fbd17eaaa1f3d95] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:04:16.604 INFO 26087 --- [ocp-schedules-11,005d5de80f48d050,2fbd17eaaa1f3d95] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:04:16.604 INFO 26087 --- [ocp-schedules-11,005d5de80f48d050,2fbd17eaaa1f3d95] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:04:16.608 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.a.service.TaskRepositoryService : fetchNeedExecute, tryAcquireTasksCount=1, fetchedCount=1 2025-04-08 21:04:16.609 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='cluster_cgroup_value', labels={app=OB, ob_cluster_id=1744114127, ob_cluster_name=ocpdb}}, value: 0, timestamp: 1744117456 : 2025-04-08 13:04:16 2025-04-08 21:04:16.612 INFO 26087 --- [ocp-schedules-14,3fab4bf9a487c598,5802257512957da5] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:04:16.612 INFO 26087 --- [ocp-schedules-14,3fab4bf9a487c598,5802257512957da5] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:04:16.620 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:04:16.626 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.alarm.core.process.AlarmProcessor : AlarmProcessor init done, lastProcessTime=2025-04-08T20:54:16.620+08:00 2025-04-08 21:04:16.627 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.alarm.core.task.AlarmProcessTask : AlarmProcessTask created 2025-04-08 21:04:16.627 ERROR 26087 --- [ocp-schedules-12,f5561ba9a1137b99,476a133023e951d8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:04:16.627 INFO 26087 --- [ocp-schedules-12,f5561ba9a1137b99,476a133023e951d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:04:16.627 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.alarm.utils.task.AlarmTaskManager : start task, taskInfo=TaskInfo(id=1, uuid=254bed2a-3c53-4ecf-b850-9c2422739b3a, taskType=AlarmProcess, target=null, parameters=null, intervalSeconds=10, status=EXECUTING, executorName=192.168.1.61:26087, createAt=2025-04-08T12:21:34.832298Z, startAt=2025-04-08T13:04:16.594Z, endAt=null, heartbeatAt=2025-04-08T13:04:16.594Z, result=executing) 2025-04-08 21:04:16.627 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.alarm.utils.task.AlarmTaskManager : startNeedExecute, activeCount=0, tryStartCount=1, startedCount=1 2025-04-08 21:04:16.633 INFO 26087 --- [ocp-schedules-30,d8a35d98406fbf00,021b0b6ab3f7f8d6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:04:16.633 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:04:16.633 INFO 26087 --- [ocp-schedules-21,8c622a7b8e880a84,387b72d73646c3c9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:04:16.647 INFO 26087 --- [ocp-schedules-30,d8a35d98406fbf00,021b0b6ab3f7f8d6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:04:16.647 INFO 26087 --- [ocp-schedules-30,d8a35d98406fbf00,021b0b6ab3f7f8d6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:04:16.672 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:04:16.670715+08:00, lastProcessTime=2025-04-08T20:54:16.620+08:00 2025-04-08 21:04:16.708 INFO 26087 --- [ocp-schedules-20,534e0e2c564191bd,dab219331b3d6c4c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:04:16.709 INFO 26087 --- [ocp-schedules-20,534e0e2c564191bd,dab219331b3d6c4c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:04:16.728 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:04:16.739 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets 2025-04-08 21:04:16.740 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:04:16.741 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:04:16.741 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:04:16.742 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:04:16.744 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:04:16.745 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:04:16.746 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.746 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:04:16.746 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:04:16.746 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:04:16.746 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:04:16.746 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:04:16.747 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:04:16.775 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:04:16.775 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:04:16.775 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.775 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:04:16.775 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:04:16.776 INFO 26087 --- [ocp-schedules-10,7f99c9194feb3384,57f832cca2d95752] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:04:16.776 INFO 26087 --- [ocp-schedules-10,7f99c9194feb3384,57f832cca2d95752] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:04:16.777 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:04:16.777 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.777 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:04:16.777 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:04:16.777 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:04:16.778 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:04:16.779 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:04:16.779 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.779 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.779 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.780 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.780 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.780 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:04:16.780 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:04:16.781 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:04:16.781 INFO 26087 --- [ocp-schedules-10,7f99c9194feb3384,57f832cca2d95752] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:04:16.781 INFO 26087 --- [ocp-schedules-10,7f99c9194feb3384,57f832cca2d95752] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:04:16.782 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:04:16.783 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:04:16.784 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:04:16.785 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:04:16.786 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-10,7f99c9194feb3384,57f832cca2d95752] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-10,7f99c9194feb3384,57f832cca2d95752] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:04:16.787 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:04:16.788 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:04:16.789 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:04:16.790 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets , current targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:04:16.791 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:04:16.792 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:04:16.793 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:04:16.805 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:04:16.806 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:04:16.807 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:04:16.808 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:04:16.809 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:04:16.810 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:04:16.811 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:04:16.812 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:04:16.813 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:04:16.814 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:04:16.815 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:04:16.816 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:04:16.817 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:04:16.670715+08:00 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:04:16.817 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:04:16.818 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:04:16.819 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:04:16.819 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.819 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:04:16.819 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:04:16.819 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:04:16.819 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:04:16.820 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:04:16.821 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets 2025-04-08 21:04:16.821 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:04:16.821 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:04:16.821 INFO 26087 --- [ocp-schedules-2,4eac35c9a9494b6c,cd8badeabf6e4d6c] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:04:16.853 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:04:16.862 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.862 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:04:16.862 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:04:16.862 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:04:16.864 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:04:16.865 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:04:16.865 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:04:16.865 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:04:16.865 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:04:16.865 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:04:16.866 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.866 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:04:16.866 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:04:16.867 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:04:16.868 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:04:16.869 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:04:16.870 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:04:16.871 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:04:16.872 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:04:16.873 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:04:16.874 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:04:16.875 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:04:16.875 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:04:16.875 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:04:16.876 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:04:16.877 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:04:16.879 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:04:16.879 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.879 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:04:16.879 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:04:16.880 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:04:16.880 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.880 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:04:16.880 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:04:16.880 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:04:16.881 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:04:16.882 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:04:16.883 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:04:16.884 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:04:16.884 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:04:16.884 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.884 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:04:16.884 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:04:16.884 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:04:16.885 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:04:16.886 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:04:16.887 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:04:16.888 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:04:16.890 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:04:16.891 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:04:16.892 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:04:16.893 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:04:16.894 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:04:16.895 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:04:16.896 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:04:16.897 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:04:16.898 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:04:16.899 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:04:16.900 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:04:16.901 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:04:16.902 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:04:16.903 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:04:16.904 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:04:16.905 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:04:16.905 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:04:16.905 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:04:16.905 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:04:16.905 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:04:16.906 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:04:16.907 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:04:16.908 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:04:16.909 INFO 26087 --- [ocp-schedules-16,164785c9a6db1982,be22ffa515929a1f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:04:17.095 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:17.095 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:18.355 INFO 26087 --- [prometheus-minute-schedule-1,3f55c749019fe5cc,449839300b78278b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:18.547 WARN 26087 --- [ocp-async-8,5dbef2e8411e8b00,ea8ffb98d9d4e122] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:18.547 INFO 26087 --- [ocp-async-1,f26310103233109b,51789d13227cbf60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:20.095 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:20.096 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:20.547 WARN 26087 --- [ocp-async-2,b1a8e460bc8bb38a,fb1b03b9522b83a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:20.547 INFO 26087 --- [ocp-async-1,0d4e2f22aa1fe154,a42c20f2deb574a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:04:21.355 INFO 26087 --- [prometheus-minute-schedule-1,6598eb8eadab2ef3,be17e8dcc8f41850] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:22.547 WARN 26087 --- [ocp-async-3,9c23da8c37778c40,b7d5d47dd186845f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:22.547 INFO 26087 --- [ocp-async-9,f3346ddecc0c1f91,ddec8dd4e13619a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:23.097 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:23.098 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:24.355 INFO 26087 --- [prometheus-minute-schedule-1,763707d01e6a7bdc,015f4388754ad86c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:24.548 WARN 26087 --- [ocp-async-8,05587a4c928662c8,566f87ede8bb60d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:24.548 INFO 26087 --- [ocp-async-4,410f4bad795a2690,db38085f239dfebd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:26.098 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:26.099 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:26.525 INFO 26087 --- [ocp-schedules-23,c956c0e6fad5c907,78e310d2a7e5be46] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:04:26.526 INFO 26087 --- [ocp-schedules-17,a322ef5b1fac82aa,0b053d7bf6cf5eb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:04:26.537 INFO 26087 --- [ocp-async-2,37ae3998a1949cb1,41124251e974dac4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:04:26.548 WARN 26087 --- [ocp-async-1,9af599472e39b7d2,612d1b0affff8797] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:26.548 INFO 26087 --- [ocp-async-6,cd4e8bbd3560dea9,c6b31e1363aba1d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:26.553 INFO 26087 --- [ocp-schedules-3,0f5f58e36e6897fd,4f5e3c320c1437a0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:04:26.554 INFO 26087 --- [ocp-schedules-3,0f5f58e36e6897fd,4f5e3c320c1437a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:26.554 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:04:26.561 INFO 26087 --- [ocp-schedules-4,5c2b6cf69d198877,1004597d66545534] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:04:26.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:04:26.625037+08:00, lastProcessTime=2025-04-08T21:04:16.670715+08:00 2025-04-08 21:04:26.787 INFO 26087 --- [ocp-schedules-29,18785b397128dc95,169b4f3f52ad49f9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:04:26.790 INFO 26087 --- [ocp-schedules-29,18785b397128dc95,169b4f3f52ad49f9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:04:26.790 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:04:26.625037+08:00 2025-04-08 21:04:26.965 INFO 26087 --- [ocp-schedules-14,8728e833a728e8cd,4ecbc8bf6491f904] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:04:27.355 INFO 26087 --- [prometheus-minute-schedule-1,be0c523840d0dc43,f1edd54bf7517c61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:28.548 WARN 26087 --- [ocp-async-6,f69abd841a3c9b8b,65cfd77579ad9db1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:28.548 INFO 26087 --- [ocp-async-9,10a62a6fd315d8fe,5db2111f47caf923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:29.099 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:29.100 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:30.355 INFO 26087 --- [prometheus-minute-schedule-1,8247301d7e8fbfda,44bceef27ca2a775] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:30.548 INFO 26087 --- [ocp-async-7,c514d0d6bccaaed5,dc0dcff723853c51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:30.548 WARN 26087 --- [ocp-async-10,eb7082a83a5d784d,0df952ac2ce61188] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:31.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:04:31.476 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/cluster, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:04:31.479 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:04:32.100 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:32.101 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:32.548 WARN 26087 --- [ocp-async-5,d157d850080cd91a,088544c683e206f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:32.548 INFO 26087 --- [ocp-async-1,efd667dfaf5cdeb3,c1ae56fa22b47860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:33.117 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,a74fab4f70001c57,61e0db01f641a464] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.11.12.102, traceId=a74fab4f70001c57, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-04-08 21:04:33.127 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,a74fab4f70001c57,61e0db01f641a464] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.11.12.102, traceId=a74fab4f70001c57, duration=19 ms] 2025-04-08 21:04:33.325 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,db9754dd5a4dee79,7c216450dffcf54e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=db9754dd5a4dee79, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:04:33.326 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,2e2919655b54df20,2d0315ff8f03d091] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=2e2919655b54df20, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:04:33.329 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,db9754dd5a4dee79,7c216450dffcf54e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=db9754dd5a4dee79, duration=10 ms] 2025-04-08 21:04:33.347 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:04:33.347 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:04:33.347 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:04:33.350 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,2e2919655b54df20,2d0315ff8f03d091] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=2e2919655b54df20, duration=33 ms] 2025-04-08 21:04:33.355 INFO 26087 --- [prometheus-minute-schedule-1,bae2a631baf7feaf,2b3fd885d5bc523d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:33.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,2a52e65eba0d9622,f4ef000f51053820] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=2a52e65eba0d9622, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:04:33.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,2a52e65eba0d9622,f4ef000f51053820] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=2a52e65eba0d9622, duration=36 ms] 2025-04-08 21:04:33.720 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,642c197cb451d0f2,5732ccecb451fb2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=642c197cb451d0f2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:04:33.720 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,babce6b676d9336a,6b62649ce42ffb3e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=babce6b676d9336a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:04:33.720 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,b864a72b20e219d2,e30a44978f99135e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=b864a72b20e219d2, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:04:33.721 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,483ff51b2469a566,182cff9c3969e289] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=483ff51b2469a566, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:04:33.722 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,b970e3848a3d3806,4c880be8ac15288b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=b970e3848a3d3806, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:04:33.746 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,1a9a13dc62865f7f,f88ddae604598e84] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=1a9a13dc62865f7f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:04:33.772 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,b864a72b20e219d2,e30a44978f99135e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=b864a72b20e219d2, duration=78 ms] 2025-04-08 21:04:33.774 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,babce6b676d9336a,6b62649ce42ffb3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=babce6b676d9336a, duration=79 ms] 2025-04-08 21:04:33.800 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,642c197cb451d0f2,5732ccecb451fb2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=642c197cb451d0f2, duration=104 ms] 2025-04-08 21:04:33.828 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,483ff51b2469a566,182cff9c3969e289] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=483ff51b2469a566, duration=134 ms] 2025-04-08 21:04:33.876 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0e9648bfd182f96f,87b30029a52fc606] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A04%3A33%2B08%3A00&endTime=2025-04-08T21%3A04%3A33%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=0e9648bfd182f96f, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:04:33+08:00,2025-04-08T21:04:33+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:04:33.898 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,0e9648bfd182f96f,87b30029a52fc606] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:04:33.899 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,0e9648bfd182f96f,87b30029a52fc606] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744113885 2025-04-08 21:04:33.909 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0e9648bfd182f96f,87b30029a52fc606] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=0e9648bfd182f96f, duration=76 ms] 2025-04-08 21:04:33.922 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,1a9a13dc62865f7f,f88ddae604598e84] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=1a9a13dc62865f7f, duration=194 ms] 2025-04-08 21:04:33.970 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,b970e3848a3d3806,4c880be8ac15288b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=b970e3848a3d3806, duration=274 ms] 2025-04-08 21:04:34.549 WARN 26087 --- [ocp-async-2,ef6a9e59e9b2bbba,2b1419441ab7ee22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:34.549 INFO 26087 --- [ocp-async-6,6d734d1ca073e7d7,0241a4f690c336f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:35.101 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:35.101 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:35.291 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,42fc97caa0c0e418,b0a9b059acf155bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=42fc97caa0c0e418, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:04:35.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,42fc97caa0c0e418,b0a9b059acf155bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=42fc97caa0c0e418, duration=5 ms] 2025-04-08 21:04:35.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,401a386c35d85a82,0bae6631b20ab7e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=401a386c35d85a82, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:04:35.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,e1523fc185c29d91,b8cefdce6b21d5e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=e1523fc185c29d91, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:04:35.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,401a386c35d85a82,0bae6631b20ab7e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=401a386c35d85a82, duration=5 ms] 2025-04-08 21:04:35.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,e1523fc185c29d91,b8cefdce6b21d5e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=e1523fc185c29d91, duration=5 ms] 2025-04-08 21:04:35.309 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:04:35.309 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:04:35.309 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:04:36.355 INFO 26087 --- [prometheus-minute-schedule-1,9098bf2b50fd75c5,2412f610556d3777] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:36.538 INFO 26087 --- [ocp-async-6,5123b6708bdacbcc,e6471261d35764c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:04:36.549 WARN 26087 --- [ocp-async-7,45acbfd9731b73c3,7755ba5d53b7a8a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:36.549 INFO 26087 --- [ocp-async-10,1b3412460d4ec437,607cb8da6fe08d2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:36.555 INFO 26087 --- [ocp-schedules-3,0f5f58e36e6897fd,4f5e3c320c1437a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:04:36.562 INFO 26087 --- [ocp-schedules-9,8878a20629c1e00a,620db0b02823ccfb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:04:36.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:04:36.625105+08:00, lastProcessTime=2025-04-08T21:04:26.625037+08:00 2025-04-08 21:04:36.635 INFO 26087 --- [ocp-schedules-20,190a109f7241b4a1,55bd570a20a73c17] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:04:36.654 INFO 26087 --- [ocp-schedules-20,190a109f7241b4a1,55bd570a20a73c17] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:04:36.654 INFO 26087 --- [ocp-schedules-20,190a109f7241b4a1,55bd570a20a73c17] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:04:36.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:04:36.625105+08:00 2025-04-08 21:04:36.794 INFO 26087 --- [ocp-schedules-30,1c8d41fa776931be,88e9e04537e54dfb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:04:36.796 INFO 26087 --- [ocp-schedules-30,1c8d41fa776931be,88e9e04537e54dfb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:04:37.035 INFO 26087 --- [ocp-schedules-25,9aff17eddc91491b,71263af4e3c3c5f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:04:38.102 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:38.103 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:38.106 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,decb7230a93baf03,4f0fbac47a202d7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=decb7230a93baf03, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:04:38.106 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,decb7230a93baf03,4f0fbac47a202d7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=decb7230a93baf03, duration=5 ms] 2025-04-08 21:04:38.286 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,dfc19023c8942751,e7c4172c258f10d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=dfc19023c8942751, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:04:38.298 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,dfc19023c8942751,e7c4172c258f10d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=dfc19023c8942751, duration=16 ms] 2025-04-08 21:04:38.549 WARN 26087 --- [ocp-async-3,70ced8233b0bc600,b19320adc0c58db9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:38.549 INFO 26087 --- [ocp-async-1,c976a1fc78c20878,aab066d5c4cbaa22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:39.355 INFO 26087 --- [prometheus-minute-schedule-1,707a53bbea0c9a78,438a279a44f8c3f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:39.522 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,11cb311ce848be07,ffcad865b82e11e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=11cb311ce848be07, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:39.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,11cb311ce848be07,ffcad865b82e11e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=11cb311ce848be07, duration=157 ms] 2025-04-08 21:04:40.002 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:04:40.002 INFO 26087 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:04:40.201 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,4b6de6fc9a7aff68,695f528fd876708c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=4b6de6fc9a7aff68, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:40.265 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,4b6de6fc9a7aff68,695f528fd876708c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=4b6de6fc9a7aff68, duration=70 ms] 2025-04-08 21:04:40.513 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,a76073039b5b9694,7561d9866cd625f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=a76073039b5b9694, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:04:40.519 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,a76073039b5b9694,7561d9866cd625f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=a76073039b5b9694, duration=9 ms] 2025-04-08 21:04:40.549 WARN 26087 --- [ocp-async-9,e734fde810d39767,2b349ba5dd5304f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:40.549 INFO 26087 --- [ocp-async-2,414c49868428d056,71e8eec339275123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:41.103 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:41.130 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:42.355 INFO 26087 --- [prometheus-minute-schedule-1,c3e2c150cd08e1d7,e81787a18d0ee405] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:42.549 WARN 26087 --- [ocp-async-6,d2640ae56d44b823,bc5477a2457921a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:42.549 INFO 26087 --- [ocp-async-7,e858555a32c2b51e,31bbf5e923d2fcd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:44.122 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,909be516071e31b2,d0e5cd1f0680ecf0] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/8/retry, client 10.11.12.102, traceId 909be516071e31b2] 2025-04-08 21:04:44.130 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:44.131 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:44.157 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,909be516071e31b2,d0e5cd1f0680ecf0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/8/retry?taskInstanceId=8, client=10.11.12.102, traceId=909be516071e31b2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.retryTask(Long), args=8,] 2025-04-08 21:04:44.215 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,909be516071e31b2,d0e5cd1f0680ecf0] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 61, current state: FAILED, new state: READY 2025-04-08 21:04:44.236 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,909be516071e31b2,d0e5cd1f0680ecf0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/8/retry client=10.11.12.102, traceId=909be516071e31b2, duration=117 ms] 2025-04-08 21:04:44.237 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,909be516071e31b2,d0e5cd1f0680ecf0] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=8, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=RETRY, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=null, startTime=2025-04-08T21:04:44.156+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 21:04:44.380 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,029ecd16ce42e486] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:RETRY, state: RUNNING 2025-04-08 21:04:44.398 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:44.444 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,788bcfc633161a22,fd86c2f99c1a67c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=788bcfc633161a22, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:04:44.445 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,99529df14df91db3,71dda69c7b39e7f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=99529df14df91db3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:44.448 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,788bcfc633161a22,fd86c2f99c1a67c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=788bcfc633161a22, duration=8 ms] 2025-04-08 21:04:44.408 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Retry subtask, id=61, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=retry, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:44.378+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=61}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:04:44.460 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,99529df14df91db3,71dda69c7b39e7f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=99529df14df91db3, duration=19 ms] 2025-04-08 21:04:44.457 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:04:44.463 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:04:44.480 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:04:44.549 WARN 26087 --- [ocp-async-5,f37db7d66c5370eb,da60078d737b7665] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:44.549 INFO 26087 --- [ocp-async-7,f7bd1c42e9305da5,3eed85acd81e6dd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:44.528 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.connector.Connectors : init global connector holder. 2025-04-08 21:04:44.599 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.i.connector.ConnectorHolder : Init connector holder, maxIdleSecs=3600 2025-04-08 21:04:44.935 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 21:04:45.062 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,3dba866ec94d6617,66dee20b9d02caf7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=3dba866ec94d6617, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:45.069 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,3dba866ec94d6617,66dee20b9d02caf7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=3dba866ec94d6617, duration=12 ms] 2025-04-08 21:04:45.080 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 21:04:45.098 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 21:04:45.251 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.61:22 success 2025-04-08 21:04:45.291 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@192.168.1.61 2025-04-08 21:04:45.294 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.61, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:1 2025-04-08 21:04:45.327 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:45.355 INFO 26087 --- [prometheus-minute-schedule-1,850609c7f6f31847,5d21b22db34ddc74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:45.355 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.388 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:04:45.423 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.426 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.430 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:45.465 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.504 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:04:45.530 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.554 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.560 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:45.606 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.646 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:04:45.678 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.687 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.695 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:04:45.698 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:45.717 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:45.757 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:04:45.810 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,d5ac6220e4a4d425,c8fdab65f29bd7e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=d5ac6220e4a4d425, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:45.823 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,d5ac6220e4a4d425,c8fdab65f29bd7e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=d5ac6220e4a4d425, duration=17 ms] 2025-04-08 21:04:45.896 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,ec14686ba02c561c,06440a67709d4295] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=ec14686ba02c561c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:45.899 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,ec14686ba02c561c,06440a67709d4295] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=ec14686ba02c561c, duration=8 ms] 2025-04-08 21:04:46.128 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.148 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.160 WARN 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:04:46.195 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:04:46.206 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:04:46.210 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:46.231 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.271 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 21:04:46.521 INFO 26087 --- [ocp-schedules-7,ad332729af243ee4,609507a94f6bdcad] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:04:46.524 INFO 26087 --- [ocp-schedules-10,f3c8d0350dc93e67,8725f48dcf332355] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:04:46.524 INFO 26087 --- [ocp-schedules-20,15ec02b6876eb2f6,47b113f661d86092] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:04:46.526 INFO 26087 --- [ocp-schedules-10,f3c8d0350dc93e67,8725f48dcf332355] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:46.526 INFO 26087 --- [ocp-schedules-20,15ec02b6876eb2f6,47b113f661d86092] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:46.526 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:04:46.527 INFO 26087 --- [ocp-schedules-11,88683fe923ead039,8768e2f869a48d17] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:04:46.538 INFO 26087 --- [ocp-async-1,cd1e9579f8ad3b3c,aaa7e357b55474ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:04:46.550 WARN 26087 --- [ocp-async-2,08c5da4f8d03ff6f,6994c76011ff7ae4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:46.550 INFO 26087 --- [ocp-async-8,f707a32a5e062a56,ab9074f35b99eea9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:46.563 INFO 26087 --- [ocp-schedules-9,14eae9699288d3dd,ad8457088ad3ec7c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:04:46.566 INFO 26087 --- [ocp-schedules-24,92d4473806b52045,c46001a698e6dfff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:04:46.566 INFO 26087 --- [ocp-schedules-24,c9dfa0ff0a749a5d,70ea132af01982ab] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:04:46.567 INFO 26087 --- [ocp-schedules-24,c9dfa0ff0a749a5d,70ea132af01982ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [2] 2025-04-08 21:04:46.572 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=2, status=TAKINGOVER 2025-04-08 21:04:46.572 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=2, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T20:41:23+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:04:46.574 INFO 26087 --- [ocp-schedules-4,7d17100fed232713,7c240c68f613aa41] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:04:46.626 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:04:46.624957+08:00, lastProcessTime=2025-04-08T21:04:36.625105+08:00 2025-04-08 21:04:46.652 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.655 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.656 WARN 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:04:46.658 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:04:46.659 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:04:46.696 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:04:46.699 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:04:46.703 WARN 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:04:46.705 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:04:46.709 WARN 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:04:46.712 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:04:46.715 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:04:46.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:04:46.624957+08:00 2025-04-08 21:04:46.727 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:46.746 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.786 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:46.803 INFO 26087 --- [ocp-schedules-15,340e3611c8c885d1,4ab0a5702e6ceb95] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:04:46.805 INFO 26087 --- [ocp-schedules-15,340e3611c8c885d1,4ab0a5702e6ceb95] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:04:46.805 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.810 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 21:04:46.860 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.863 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.912 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:46.942 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.949 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 21:04:46.989 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:46.992 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:47.014 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:04:47.032 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:47.064 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:47.068 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 21:04:47.091 INFO 26087 --- [ocp-schedules-29,aad6b12692f01623,1627ad77c08df0f2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:04:47.112 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,647c8a485771fa91,3e177b03762f7074] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=647c8a485771fa91, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:47.109 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:04:47.114 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:04:47.119 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:04:47.125 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,647c8a485771fa91,3e177b03762f7074] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=647c8a485771fa91, duration=20 ms] 2025-04-08 21:04:47.128 ERROR 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:04:47.131 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:47.132 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:47.135 ERROR 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:04:47.192 WARN 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.runtime.Subtask.retry(Subtask.java:49) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.retry(JavaSubtaskRunner.java:76) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:35) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:04:47.202 INFO 26087 --- [manual-subtask-executor1,24f1d56478c4d70e,4a3b631773f50e6e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:RETRY, state: FAILED 2025-04-08 21:04:47.211 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,78bdf1ed32793fe6,d766f410a07da123] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=78bdf1ed32793fe6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:47.214 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,78bdf1ed32793fe6,d766f410a07da123] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=78bdf1ed32793fe6, duration=7 ms] 2025-04-08 21:04:47.292 INFO 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=8, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=2, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, createTime=2025-04-08T20:41:23+08:00, startTime=2025-04-08T20:41:23.075+08:00, endTime=2025-04-08T20:44:41.716+08:00, subtasks=[SubtaskInstanceEntity{id=76, name=Update OB cluster status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.obops.internal.cluster.task.UpdateObClusterStatusTask, seriesId=9, startTime=null, endTime=null}, SubtaskInstanceEntity{id=81, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=18, startTime=2025-04-08T20:41:25.317+08:00, endTime=2025-04-08T20:41:25.493+08:00}, SubtaskInstanceEntity{id=46, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=66, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=34, startTime=null, endTime=null}, SubtaskInstanceEntity{id=68, name=Active agent modules, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obagent.ActiveAgentModuleTask, seriesId=24, startTime=null, endTime=null}, SubtaskInstanceEntity{id=84, name=Wait node, state=PE 2025-04-08 21:04:47.301 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:04:47.301 INFO 26087 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:8, status:FAILED 2025-04-08 21:04:48.355 INFO 26087 --- [prometheus-minute-schedule-1,7e601757a5feb530,8305653dd300fde7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:48.432 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8e3f36726895f169,e137cd0e7fcee716] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=8e3f36726895f169, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:48.446 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8e3f36726895f169,e137cd0e7fcee716] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=8e3f36726895f169, duration=18 ms] 2025-04-08 21:04:48.545 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,a58c60d55dab91cc,3ebaadcaadfd547f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=a58c60d55dab91cc, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:48.548 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,a58c60d55dab91cc,3ebaadcaadfd547f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=a58c60d55dab91cc, duration=7 ms] 2025-04-08 21:04:48.550 WARN 26087 --- [ocp-async-6,6d3be261fab84a76,67a8d711afb77f4a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:48.550 INFO 26087 --- [ocp-async-4,6cbcf1a4caef7451,49d8287d5eb2d2f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:48.762 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,06ce5f368621a4a6,f9ded7b42c954f8b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=06ce5f368621a4a6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:48.765 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,06ce5f368621a4a6,f9ded7b42c954f8b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=06ce5f368621a4a6, duration=7 ms] 2025-04-08 21:04:50.132 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:50.132 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:50.550 WARN 26087 --- [ocp-async-5,a262301cfe332c80,73e453199bbd38b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:50.550 INFO 26087 --- [ocp-async-3,ad6c44dfc49ec4dd,2a487fbca7cbc677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:51.355 INFO 26087 --- [prometheus-minute-schedule-1,6fe8ea078d6d3b16,ae41a2982725715a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:52.462 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,dd8e083163885a4f,8b921f85cfefccc0] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/8/rollback, client 10.11.12.102, traceId dd8e083163885a4f] 2025-04-08 21:04:52.465 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,dd8e083163885a4f,8b921f85cfefccc0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/8/rollback?taskInstanceId=8, client=10.11.12.102, traceId=dd8e083163885a4f, method=NoDataResponse com.oceanbase.ocp.server.common.controller.TaskController.rollbackTask(Long), args=8,] 2025-04-08 21:04:52.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,dd8e083163885a4f,8b921f85cfefccc0] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 61, current state: FAILED, new state: READY 2025-04-08 21:04:52.550 WARN 26087 --- [ocp-async-9,72b11781905d58d1,a3237fae85c88b34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:52.550 INFO 26087 --- [ocp-async-2,b5a05c03957465d6,74b462b76b5b1c18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:52.550 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,dd8e083163885a4f,8b921f85cfefccc0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/8/rollback client=10.11.12.102, traceId=dd8e083163885a4f, duration=90 ms] 2025-04-08 21:04:52.757 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,f87d687a30dbefc3,4f570c0c2f9d4359] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=f87d687a30dbefc3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:52.770 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,f87d687a30dbefc3,4f570c0c2f9d4359] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=f87d687a30dbefc3, duration=17 ms] 2025-04-08 21:04:53.133 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:53.133 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:53.183 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,15fd1ce70d580180,88b3ac337ef02ef5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=15fd1ce70d580180, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:53.189 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,15fd1ce70d580180,88b3ac337ef02ef5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=15fd1ce70d580180, duration=10 ms] 2025-04-08 21:04:53.401 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,9714292c55d04955] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=61, original=READY 2025-04-08 21:04:53.403 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,9714292c55d04955] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:04:53.417 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:53.424 WARN 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=61, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:53.400+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=61}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:04:53.428 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:04:53.435 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:04:53.439 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:04:53.471 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:53.519 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.528 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:53.546 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.586 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:04:53.615 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.622 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.629 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:53.647 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.687 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:04:53.720 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.728 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.732 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:53.758 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.799 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:04:53.826 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.829 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.832 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:04:53.834 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:53.867 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:53.907 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:04:54.095 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,522c556b51b7a47c,4bd0c77461051c72] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=522c556b51b7a47c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:54.113 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,522c556b51b7a47c,4bd0c77461051c72] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=522c556b51b7a47c, duration=22 ms] 2025-04-08 21:04:54.279 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:54.305 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,a6b7a79c0d68cbcb,0048250b6402488d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=a6b7a79c0d68cbcb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:54.308 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,a6b7a79c0d68cbcb,0048250b6402488d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=a6b7a79c0d68cbcb, duration=8 ms] 2025-04-08 21:04:54.302 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:54.309 WARN 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:04:54.313 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:04:54.319 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:04:54.329 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:04:54.355 INFO 26087 --- [prometheus-minute-schedule-1,134b61aee7407cc9,f65a8c6c07a323c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:54.348 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:54.389 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 21:04:54.550 WARN 26087 --- [ocp-async-10,facf8494bb66845f,07d70f8e9de8017a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:54.550 INFO 26087 --- [ocp-async-6,a50a1ca5476fdf2b,fe5bdd8997d25b97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:54.775 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:54.800 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:04:54.808 WARN 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:04:54.814 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:04:54.816 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:04:54.829 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 61, operation:ROLLBACK, state: PENDING 2025-04-08 21:04:54.839 INFO 26087 --- [manual-subtask-executor2,24f1d56478c4d70e,0fed6bd9363a0a46] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[58] 2025-04-08 21:04:55.313 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,4b959a5ba2fe4fd8,9f5ee5e29b35c1dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=4b959a5ba2fe4fd8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:55.324 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,4b959a5ba2fe4fd8,9f5ee5e29b35c1dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=4b959a5ba2fe4fd8, duration=15 ms] 2025-04-08 21:04:55.410 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,06484f3f67712a3c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=58, original=READY 2025-04-08 21:04:55.412 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,06484f3f67712a3c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 58, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:04:55.464 INFO 26087 --- [manual-subtask-executor3,24f1d56478c4d70e,dbf3c30646f7fb2f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:55.467 WARN 26087 --- [manual-subtask-executor3,24f1d56478c4d70e,dbf3c30646f7fb2f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=58, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:55.410+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=58}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:04:55.505 INFO 26087 --- [manual-subtask-executor3,24f1d56478c4d70e,dbf3c30646f7fb2f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 58, operation:ROLLBACK, state: PENDING 2025-04-08 21:04:55.507 INFO 26087 --- [manual-subtask-executor3,24f1d56478c4d70e,dbf3c30646f7fb2f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[56] 2025-04-08 21:04:55.517 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,028ce2f40767150c,4743bef3c4d2a498] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments?taskInstanceId=8&subtaskInstanceId=61, client=10.11.12.102, traceId=028ce2f40767150c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=8,61,] 2025-04-08 21:04:55.520 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,028ce2f40767150c,4743bef3c4d2a498] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8/subtasks/61/logSegments client=10.11.12.102, traceId=028ce2f40767150c, duration=7 ms] 2025-04-08 21:04:56.134 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:56.134 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:56.419 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,2d212d1b8ec64433] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=56, original=READY 2025-04-08 21:04:56.421 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,2d212d1b8ec64433] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 56, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:04:56.512 INFO 26087 --- [manual-subtask-executor4,24f1d56478c4d70e,5fc7d3b9a2da23e3] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:56.527 INFO 26087 --- [ocp-schedules-20,15ec02b6876eb2f6,47b113f661d86092] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:04:56.527 INFO 26087 --- [ocp-schedules-10,f3c8d0350dc93e67,8725f48dcf332355] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:04:56.533 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,b6ba41afe50e8d02,da655cbe5b3ccf91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=b6ba41afe50e8d02, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:56.538 INFO 26087 --- [ocp-async-6,1c943d4fd7337486,ff43f03408e7337d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:04:56.544 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,b6ba41afe50e8d02,da655cbe5b3ccf91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=b6ba41afe50e8d02, duration=15 ms] 2025-04-08 21:04:56.550 WARN 26087 --- [ocp-async-7,b91169f0c3cb2895,96b89204a019fc13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:56.550 INFO 26087 --- [ocp-async-7,5dcafcb723636716,bf6d2671d45864a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:56.564 INFO 26087 --- [ocp-schedules-9,2faed8cf2913ec55,90328be6a1a83fe1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:04:56.516 WARN 26087 --- [manual-subtask-executor4,24f1d56478c4d70e,5fc7d3b9a2da23e3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=56, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:56.419+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=56}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:04:56.615 INFO 26087 --- [manual-subtask-executor4,24f1d56478c4d70e,5fc7d3b9a2da23e3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 56, operation:ROLLBACK, state: PENDING 2025-04-08 21:04:56.617 INFO 26087 --- [manual-subtask-executor4,24f1d56478c4d70e,5fc7d3b9a2da23e3] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[82] 2025-04-08 21:04:56.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:04:56.624256+08:00, lastProcessTime=2025-04-08T21:04:46.624957+08:00 2025-04-08 21:04:56.656 INFO 26087 --- [ocp-schedules-25,f13decf232a1e7a7,d93b20a4053c09d0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:04:56.661 INFO 26087 --- [ocp-schedules-25,f13decf232a1e7a7,d93b20a4053c09d0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:04:56.661 INFO 26087 --- [ocp-schedules-25,f13decf232a1e7a7,d93b20a4053c09d0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:04:56.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:04:56.624256+08:00 2025-04-08 21:04:56.809 INFO 26087 --- [ocp-schedules-11,74a63b8e80ac83db,d91ded00c2fd0725] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:04:56.811 INFO 26087 --- [ocp-schedules-11,74a63b8e80ac83db,d91ded00c2fd0725] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:04:57.147 INFO 26087 --- [ocp-schedules-5,1890189b2b6b4c66,09cd796e2d4bb2a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:04:57.355 INFO 26087 --- [prometheus-minute-schedule-1,6d0025b60fe184a4,c52bd664cf600820] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117500, interval=60 2025-04-08 21:04:57.462 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4223192c2145ac60] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=82, original=READY 2025-04-08 21:04:57.464 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4223192c2145ac60] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 82, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:04:57.507 INFO 26087 --- [manual-subtask-executor5,24f1d56478c4d70e,cd9ba7775921b602] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:57.511 WARN 26087 --- [manual-subtask-executor5,24f1d56478c4d70e,cd9ba7775921b602] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=82, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:57.462+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=82}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:04:57.521 INFO 26087 --- [manual-subtask-executor5,24f1d56478c4d70e,cd9ba7775921b602] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 82, operation:ROLLBACK, state: PENDING 2025-04-08 21:04:57.523 INFO 26087 --- [manual-subtask-executor5,24f1d56478c4d70e,cd9ba7775921b602] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[62] 2025-04-08 21:04:57.804 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,235e409ffbcb55c1,f48fc3e7a7a27f82] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=235e409ffbcb55c1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:57.814 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,235e409ffbcb55c1,f48fc3e7a7a27f82] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=235e409ffbcb55c1, duration=15 ms] 2025-04-08 21:04:58.472 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4a6809ed8d9adb3c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=62, original=READY 2025-04-08 21:04:58.475 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4a6809ed8d9adb3c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 62, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:04:58.513 INFO 26087 --- [manual-subtask-executor6,24f1d56478c4d70e,56248dba9c61e612] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:58.518 WARN 26087 --- [manual-subtask-executor6,24f1d56478c4d70e,56248dba9c61e612] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=62, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:58.472+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=62}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:04:58.522 INFO 26087 --- [manual-subtask-executor6,24f1d56478c4d70e,56248dba9c61e612] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 21:04:58.535 INFO 26087 --- [manual-subtask-executor6,24f1d56478c4d70e,56248dba9c61e612] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 62, operation:ROLLBACK, state: PENDING 2025-04-08 21:04:58.537 INFO 26087 --- [manual-subtask-executor6,24f1d56478c4d70e,56248dba9c61e612] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[50] 2025-04-08 21:04:58.550 WARN 26087 --- [ocp-async-1,9a8d719820078742,327f280a2d3b5f07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:04:58.550 INFO 26087 --- [ocp-async-7,0684cab58716865b,85c99be24b005b42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:04:59.015 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,d9bc727b77d7a6b4,ee866d621971f9e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=d9bc727b77d7a6b4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:04:59.027 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,d9bc727b77d7a6b4,ee866d621971f9e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=d9bc727b77d7a6b4, duration=17 ms] 2025-04-08 21:04:59.134 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:04:59.135 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:04:59.509 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,2a11ef70a4eb1c24] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=50, original=READY 2025-04-08 21:04:59.511 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,2a11ef70a4eb1c24] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 50, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:04:59.520 INFO 26087 --- [manual-subtask-executor7,24f1d56478c4d70e,9e31bd1588d3dd6c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:04:59.523 WARN 26087 --- [manual-subtask-executor7,24f1d56478c4d70e,9e31bd1588d3dd6c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=50, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:04:59.509+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=50}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:04:59.590 INFO 26087 --- [manual-subtask-executor7,24f1d56478c4d70e,9e31bd1588d3dd6c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 50, operation:ROLLBACK, state: PENDING 2025-04-08 21:04:59.592 INFO 26087 --- [manual-subtask-executor7,24f1d56478c4d70e,9e31bd1588d3dd6c] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[65] 2025-04-08 21:05:00.206 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:03:27.0, nextRun=2025-04-08 21:05:00.0, currentDate=Tue Apr 08 21:05:00 CST 2025 2025-04-08 21:05:00.206 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:03:27.0, nextRunTime=2025-04-08 21:05:00.0} 2025-04-08 21:05:00.210 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:05:00.210 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:05:00.210+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:05:00.218 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:03:27.0, nextRun=2025-04-08 21:05:00.0, currentDate=Tue Apr 08 21:05:00 CST 2025 2025-04-08 21:05:00.218 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:03:27.0, nextRunTime=2025-04-08 21:05:00.0} 2025-04-08 21:05:00.223 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:05:00.225 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,6a709e9a6d4c657b,4d0b66f0c991f19e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=6a709e9a6d4c657b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:00.226 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:05:00.226 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,778a3831e64de445] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:05:00.236 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,6a709e9a6d4c657b,4d0b66f0c991f19e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=6a709e9a6d4c657b, duration=15 ms] 2025-04-08 21:05:00.355 INFO 26087 --- [prometheus-minute-schedule-1,068de4b266248005,9d6f25fe26b02e5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:00.517 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b64f7c4b0a9dd7bc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=65, original=READY 2025-04-08 21:05:00.519 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b64f7c4b0a9dd7bc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 65, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:05:00.537 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b64f7c4b0a9dd7bc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000004, operation:EXECUTE, state: RUNNING 2025-04-08 21:05:00.551 WARN 26087 --- [ocp-async-8,473b2abe75ab21b4,c81751cc1c8d0c2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:00.551 INFO 26087 --- [ocp-async-7,c3289f550632696f,101ce71d695df8cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:00.540 INFO 26087 --- [manual-subtask-executor8,24f1d56478c4d70e,797ad6209f0e323c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:05:00.597 WARN 26087 --- [manual-subtask-executor8,24f1d56478c4d70e,797ad6209f0e323c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=65, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:05:00.517+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=65}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:05:00.601 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:05:00.604 INFO 26087 --- [manual-subtask-executor8,24f1d56478c4d70e,797ad6209f0e323c] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 21:05:00.605 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000004, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000005, task_operation=execute, latest_execution_start_time=2025-04-08T21:05:00.536+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000004}, listMap={}}, executor=192.168.1.61 2025-04-08 21:05:00.623 INFO 26087 --- [manual-subtask-executor8,24f1d56478c4d70e,797ad6209f0e323c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 65, operation:ROLLBACK, state: PENDING 2025-04-08 21:05:00.623 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [manual-subtask-executor8,24f1d56478c4d70e,797ad6209f0e323c] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[73] 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.625 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:05:00.633 INFO 26087 --- [subtask-executor4,8ec1a44aeefd3a62,28ff2c962bc38bf5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000004, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:05:01.258 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000005, status:SUCCESS 2025-04-08 21:05:01.259 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000005, status:SUCCESS 2025-04-08 21:05:01.447 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,6c3230da5f920890,030d3d9d42fdd192] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=6c3230da5f920890, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:01.457 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,6c3230da5f920890,030d3d9d42fdd192] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=6c3230da5f920890, duration=15 ms] 2025-04-08 21:05:01.561 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,e8b62e09f47f4455] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=73, original=READY 2025-04-08 21:05:01.563 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,e8b62e09f47f4455] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 73, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:05:01.600 INFO 26087 --- [manual-subtask-executor9,24f1d56478c4d70e,e0c8223625f135f1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:05:01.605 WARN 26087 --- [manual-subtask-executor9,24f1d56478c4d70e,e0c8223625f135f1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=73, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:05:01.560+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=73}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:05:01.619 INFO 26087 --- [manual-subtask-executor9,24f1d56478c4d70e,e0c8223625f135f1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 73, operation:ROLLBACK, state: PENDING 2025-04-08 21:05:01.621 INFO 26087 --- [manual-subtask-executor9,24f1d56478c4d70e,e0c8223625f135f1] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[47] 2025-04-08 21:05:02.135 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:02.136 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:02.551 WARN 26087 --- [ocp-async-9,06558052e43cbe24,7d4ce2da16736b23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:02.551 INFO 26087 --- [ocp-async-7,926c4e56797afd5c,410ce5578bdee5ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:02.582 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,8b7bf333539a19cc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=47, original=READY 2025-04-08 21:05:02.584 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,8b7bf333539a19cc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 47, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:05:02.602 INFO 26087 --- [manual-subtask-executor10,24f1d56478c4d70e,13eb3f60aba846c4] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:05:02.606 WARN 26087 --- [manual-subtask-executor10,24f1d56478c4d70e,13eb3f60aba846c4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=47, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:05:02.582+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=47}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:05:02.620 INFO 26087 --- [manual-subtask-executor10,24f1d56478c4d70e,13eb3f60aba846c4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 47, operation:ROLLBACK, state: PENDING 2025-04-08 21:05:02.621 INFO 26087 --- [manual-subtask-executor10,24f1d56478c4d70e,13eb3f60aba846c4] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[81] 2025-04-08 21:05:02.648 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,7157c61b91aaaccd,b6ab45b9a64c063f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=7157c61b91aaaccd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:02.658 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,7157c61b91aaaccd,b6ab45b9a64c063f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=7157c61b91aaaccd, duration=15 ms] 2025-04-08 21:05:03.355 INFO 26087 --- [prometheus-minute-schedule-1,1870e1a4ffffaa1f,e55694c2fa8088c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:03.600 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,689330567db64b1d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=81, original=READY 2025-04-08 21:05:03.602 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,689330567db64b1d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 81, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:05:03.610 INFO 26087 --- [manual-subtask-executor11,24f1d56478c4d70e,d366db964eea5da1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:05:03.616 WARN 26087 --- [manual-subtask-executor11,24f1d56478c4d70e,d366db964eea5da1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=81, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:05:03.600+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=81}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:05:03.633 INFO 26087 --- [manual-subtask-executor11,24f1d56478c4d70e,d366db964eea5da1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 81, operation:ROLLBACK, state: PENDING 2025-04-08 21:05:03.635 INFO 26087 --- [manual-subtask-executor11,24f1d56478c4d70e,d366db964eea5da1] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[71] 2025-04-08 21:05:03.854 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,f95f1652db41e120,3399c7706cf09811] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=f95f1652db41e120, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:03.864 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,f95f1652db41e120,3399c7706cf09811] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=f95f1652db41e120, duration=14 ms] 2025-04-08 21:05:04.551 WARN 26087 --- [ocp-async-6,f38166d7c3ced632,8f6a05f976feee39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:04.551 INFO 26087 --- [ocp-async-3,a33eea98ea078cf8,2b9d1e901ea5b070] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:04.607 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,8025aba88893929f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=71, original=READY 2025-04-08 21:05:04.609 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,8025aba88893929f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 71, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:05:04.625 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:05:04.630 WARN 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=71, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:05:04.607+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=71}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:05:04.635 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=2 2025-04-08 21:05:04.644 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-2, method=deleteHost 2025-04-08 21:05:04.717 WARN 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 2, please retry manually 2025-04-08 21:05:04.722 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=2 2025-04-08 21:05:04.726 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-04-08 21:05:04.737 WARN 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=2 2025-04-08 21:05:04.740 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 2 2025-04-08 21:05:04.761 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [2] 2025-04-08 21:05:04.774 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [2] 2025-04-08 21:05:04.797 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [2] 2025-04-08 21:05:04.825 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [2] 2025-04-08 21:05:04.842 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [2:192.168.1.61], from status [NEW] to [NEW] 2025-04-08 21:05:04.919 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 71, operation:ROLLBACK, state: PENDING 2025-04-08 21:05:04.921 INFO 26087 --- [manual-subtask-executor12,24f1d56478c4d70e,a63b51cc18e8c89f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[60] 2025-04-08 21:05:05.104 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,9e423540e0696b36,6e70c7fa8591cf4e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=9e423540e0696b36, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:05.115 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,9e423540e0696b36,6e70c7fa8591cf4e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=9e423540e0696b36, duration=16 ms] 2025-04-08 21:05:05.136 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:05.137 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:05.618 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4cb6bb3d0a147529] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=60, original=READY 2025-04-08 21:05:05.620 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4cb6bb3d0a147529] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 60, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:05:05.634 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:05:05.638 WARN 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=60, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=8, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=2, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:05:05.618+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=60}, listMap={add_region_ids=[3], server_ids=[2], add_idc_ids=[2], all_host_ids=[2], add_host_ids=[2], host_ids=[2], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:05:05.639 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster metadata 2025-04-08 21:05:05.654 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.o.i.c.l.CredentialServiceListener : Detect cluster been deleted, begin to delete credentials of this cluster, clusterId=2 2025-04-08 21:05:05.682 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.c.CredentialServiceImpl : Deleting credential: 6 2025-04-08 21:05:05.713 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.c.CredentialServiceImpl : Delete credential 6 success 2025-04-08 21:05:05.721 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.o.i.c.l.CredentialServiceListener : Delete credentials success, clusterId=2 2025-04-08 21:05:05.723 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.o.i.c.l.ClusterServiceListener : detect cluster been deleted, begin to delete parameter change and compaction info, clusterId=2 2025-04-08 21:05:05.818 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000001, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:05:05.824 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.o.i.c.l.ClusterServiceListener : detect parameter change and compaction info finished, clusterId=2 2025-04-08 21:05:05.831 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000002, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["ocpdb"]} 2025-04-08 21:05:05.840 ERROR 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 2] 2025-04-08 21:05:05.860 WARN 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,2 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$5d1537be.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:05:05.915 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.o.internal.ObproxyServiceListener : detect cluster been deleted, begin to delete oblinks, obclusterName=ocpdb, obClusterId=1744114127 2025-04-08 21:05:05.926 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000003, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:05:05.937 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.b.i.l.BackupServiceListener : detect cluster been deleted, begin to delete backup strategy, clusterId=2 2025-04-08 21:05:05.953 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.b.i.l.BackupServiceListener : detect backup strategy finished, clusterId=2 2025-04-08 21:05:05.994 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.s.o.OrganizationResourceManager : delete resource for current organization,organizationId=10000000,resourceType=2, resourceId=2 2025-04-08 21:05:06.022 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster finished 2025-04-08 21:05:06.025 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host started, hostIds=[2] 2025-04-08 21:05:06.033 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-2, method=deleteHost 2025-04-08 21:05:06.042 ERROR 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 2] 2025-04-08 21:05:06.063 WARN 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 2 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,2 2025-04-08 21:05:06.069 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host finished 2025-04-08 21:05:06.093 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.host.service.IdcServiceImpl : Deleted idc: id=2 2025-04-08 21:05:06.134 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.host.service.RegionServiceImpl : Deleted region: 3 deault_region 2025-04-08 21:05:06.141 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : rollback finished 2025-04-08 21:05:06.161 INFO 26087 --- [manual-subtask-executor13,24f1d56478c4d70e,e8a35b7735574a20] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 60, operation:ROLLBACK, state: PENDING 2025-04-08 21:05:06.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,7ae06c3e88b44b92,f75545d23e84901b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=7ae06c3e88b44b92, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:06.325 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,7ae06c3e88b44b92,f75545d23e84901b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=7ae06c3e88b44b92, duration=14 ms] 2025-04-08 21:05:06.355 INFO 26087 --- [prometheus-minute-schedule-1,49084e6c306055a4,8444512d56607c44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:06.355 INFO 26087 --- [prometheus-cache-monitor1,26c704dd3e74b770,c14bece6d3047149] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":0,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":0} 2025-04-08 21:05:06.355 INFO 26087 --- [prometheus-cache-monitor1,26c704dd3e74b770,c14bece6d3047149] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:05:06.414 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 21:05:06.415 INFO 26087 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 21:05:06.417 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:05:06.418 WARN 26087 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster ocpdb not found in config server 2025-04-08 21:05:06.424 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000001, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:05:06.425 INFO 26087 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Receive invalidate ob version event, clusters:[ocpdb] 2025-04-08 21:05:06.428 INFO 26087 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Invalidate ob version success, clusters:[ocpdb] 2025-04-08 21:05:06.430 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000002, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["ocpdb"]} 2025-04-08 21:05:06.430 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 21:05:06.431 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:05:06.431 INFO 26087 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 21:05:06.432 WARN 26087 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster ocpdb not found in config server 2025-04-08 21:05:06.433 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000003, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:05:06.538 INFO 26087 --- [ocp-async-3,3832c293197f1d94,affd1d446c426145] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:05:06.551 WARN 26087 --- [ocp-async-8,7e36d8cd4e3c5bf0,bf8284228f0991ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:06.551 INFO 26087 --- [ocp-async-4,4bc4215d4c2fefe2,92043fdf684fb775] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:06.565 INFO 26087 --- [ocp-schedules-17,e0b667940d5af433,e70549d455e8289f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:05:06.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:05:06.624893+08:00, lastProcessTime=2025-04-08T21:04:56.624256+08:00 2025-04-08 21:05:06.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb 2025-04-08 21:05:06.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:05:06.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb 2025-04-08 21:05:06.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:05:06.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:05:06.740 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:05:06.624893+08:00 2025-04-08 21:05:06.815 INFO 26087 --- [ocp-schedules-15,ef21b2a7c0d1906b,d7036976be05aca6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:05:06.816 INFO 26087 --- [ocp-schedules-15,ef21b2a7c0d1906b,d7036976be05aca6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:05:07.010 ERROR 26087 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : GetClusterOverview, clusterId:2, NotFoundException {} com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=2 at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl.nullSafeGetClusterOverview(ClusterDaoManagerImpl.java:94) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$FastClassBySpringCGLIB$$1f10cbd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$EnhancerBySpringCGLIB$$5edf77dc.nullSafeGetClusterOverview() at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:126) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:05:07.011 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:05:07.189 INFO 26087 --- [ocp-schedules-29,2bf53350a43ca43e,d53c0865fcb80e48] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:05:07.336 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000003 2025-04-08 21:05:07.336 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000002 2025-04-08 21:05:07.336 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000001 2025-04-08 21:05:07.524 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,a0b98ba06c62738c,4738c93f84c6453d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/8?taskInstanceId=8, client=10.11.12.102, traceId=a0b98ba06c62738c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=8,] 2025-04-08 21:05:07.534 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,a0b98ba06c62738c,4738c93f84c6453d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/8 client=10.11.12.102, traceId=a0b98ba06c62738c, duration=15 ms] 2025-04-08 21:05:07.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,aad6c5ac5bed19aa,1c4043707021380f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=aad6c5ac5bed19aa, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:05:07.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47ad70be818e31ec,dbdc3b890234aadb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=47ad70be818e31ec, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:05:07.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,aad6c5ac5bed19aa,1c4043707021380f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=aad6c5ac5bed19aa, duration=4 ms] 2025-04-08 21:05:07.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,90315f81217bd650,1f6a819c8fd09c64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=90315f81217bd650, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:05:07.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,835c1e4a0320264b,66a2279f1de8b460] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=835c1e4a0320264b, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:05:07.777 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,835c1e4a0320264b,66a2279f1de8b460] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=835c1e4a0320264b, duration=5 ms] 2025-04-08 21:05:07.777 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,5bf0b5e900e0882f,f6762b2d80c87fe0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A05%3A07%2B08%3A00&endTime=2025-04-08T21%3A05%3A07%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=5bf0b5e900e0882f, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:05:07+08:00,2025-04-08T21:05:07+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:05:07.780 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47ad70be818e31ec,dbdc3b890234aadb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=47ad70be818e31ec, duration=8 ms] 2025-04-08 21:05:07.781 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-4,5bf0b5e900e0882f,f6762b2d80c87fe0] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:05:07.781 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-4,5bf0b5e900e0882f,f6762b2d80c87fe0] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744113915 2025-04-08 21:05:07.783 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,5bf0b5e900e0882f,f6762b2d80c87fe0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=5bf0b5e900e0882f, duration=11 ms] 2025-04-08 21:05:07.783 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,90315f81217bd650,1f6a819c8fd09c64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=90315f81217bd650, duration=11 ms] 2025-04-08 21:05:07.796 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,f59af6ee446bd0c2,5565cc56aa9b2c2d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=f59af6ee446bd0c2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:05:07.801 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,f59af6ee446bd0c2,5565cc56aa9b2c2d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=f59af6ee446bd0c2, duration=10 ms] 2025-04-08 21:05:08.137 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:08.138 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:08.551 WARN 26087 --- [ocp-async-7,f991c768891b3a87,827495ed2bcb26cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:08.551 INFO 26087 --- [ocp-async-9,1436b12d08366e55,bf30061d10bd15e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:08.749 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,b30537096f87aa60,a3f4d1d71bb73212] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=b30537096f87aa60, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:05:08.752 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,b30537096f87aa60,a3f4d1d71bb73212] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=b30537096f87aa60, duration=6 ms] 2025-04-08 21:05:09.355 INFO 26087 --- [prometheus-minute-schedule-1,f772211f69a36498,9c2bf60f385f7e7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:10.001 INFO 26087 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:05:10.001 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:05:10.143 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:05:10.144 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/cluster, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:05:10.144 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:05:10.384 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,007d05731a9e1454,042dab62bbe65e19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.11.12.102, traceId=007d05731a9e1454, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-04-08 21:05:10.385 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,007d05731a9e1454,042dab62bbe65e19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.11.12.102, traceId=007d05731a9e1454, duration=5 ms] 2025-04-08 21:05:10.551 WARN 26087 --- [ocp-async-1,e7cbd39cba00fb4d,97be5a12f7373840] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:10.551 INFO 26087 --- [ocp-async-10,175e719fc9b1ef68,4986cf6da8ca0f53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:10.603 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:05:10.617 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,7e13043519465d52,70ee4e3a8bd5558e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=7e13043519465d52, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:05:10.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,237aa94c29917a8a,1b90ad1945579f6f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=237aa94c29917a8a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:05:10.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,237aa94c29917a8a,1b90ad1945579f6f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=237aa94c29917a8a, duration=4 ms] 2025-04-08 21:05:10.620 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,7e13043519465d52,70ee4e3a8bd5558e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=7e13043519465d52, duration=6 ms] 2025-04-08 21:05:10.629 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:05:10.629 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:05:10.629 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:05:10.795 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,7ee0b95689dca4fa,bf5fb2fc6691c5de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=7ee0b95689dca4fa, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:05:10.849 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,845d5f3f576da501,c302c8f65d8a5f83] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=845d5f3f576da501, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:05:10.858 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,845d5f3f576da501,c302c8f65d8a5f83] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=845d5f3f576da501, duration=12 ms] 2025-04-08 21:05:10.911 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,7ee0b95689dca4fa,bf5fb2fc6691c5de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=7ee0b95689dca4fa, duration=121 ms] 2025-04-08 21:05:10.957 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,474c35024d7bf521,17f9c2fc2cbe8a69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=474c35024d7bf521, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:05:10.957 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,1996083cde9ba05e,c5749788668ad73c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=1996083cde9ba05e, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:05:10.957 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,c4ca05c0f4ceecf9,a7fdc2a032c802c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=c4ca05c0f4ceecf9, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:05:10.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,474c35024d7bf521,17f9c2fc2cbe8a69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=474c35024d7bf521, duration=5 ms] 2025-04-08 21:05:10.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,ff456afd546849b0,e9875b8e279bdf32] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=ff456afd546849b0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:05:10.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,c4ca05c0f4ceecf9,a7fdc2a032c802c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=c4ca05c0f4ceecf9, duration=5 ms] 2025-04-08 21:05:10.960 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,1996083cde9ba05e,c5749788668ad73c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=1996083cde9ba05e, duration=7 ms] 2025-04-08 21:05:10.962 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,ff456afd546849b0,e9875b8e279bdf32] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=ff456afd546849b0, duration=9 ms] 2025-04-08 21:05:11.006 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,e41143e1301ed1a6,59d127b47c42d000] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=e41143e1301ed1a6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:05:11.011 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,e41143e1301ed1a6,59d127b47c42d000] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=e41143e1301ed1a6, duration=33 ms] 2025-04-08 21:05:11.114 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,7034a716fc488ee5,e202015bc57cdf38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A05%3A11%2B08%3A00&endTime=2025-04-08T21%3A05%3A11%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=7034a716fc488ee5, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:05:11+08:00,2025-04-08T21:05:11+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:05:11.115 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-5,7034a716fc488ee5,e202015bc57cdf38] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:05:11.115 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-5,7034a716fc488ee5,e202015bc57cdf38] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744113915 2025-04-08 21:05:11.116 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,7034a716fc488ee5,e202015bc57cdf38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=7034a716fc488ee5, duration=6 ms] 2025-04-08 21:05:11.138 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:11.138 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:12.355 INFO 26087 --- [prometheus-minute-schedule-1,f52b12d31d1b3030,aa78ae9555ca19c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:12.552 WARN 26087 --- [ocp-async-3,6b6142f5536c2755,0437492782801a23] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:12.552 INFO 26087 --- [ocp-async-2,3efcee86f2b86e5d,e5cdaeb8d81d81a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:12.599 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,912ffa3a2bcddca8,2e99299bffb4b12a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=912ffa3a2bcddca8, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:05:12.599 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,27953941418bdc93,97f255bdaa888023] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=27953941418bdc93, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:05:12.599 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,912ffa3a2bcddca8,2e99299bffb4b12a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=912ffa3a2bcddca8, duration=3 ms] 2025-04-08 21:05:12.599 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,27953941418bdc93,97f255bdaa888023] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=27953941418bdc93, duration=3 ms] 2025-04-08 21:05:12.600 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,00f646d78c957baf,e759e5a339c2c008] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=00f646d78c957baf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:05:12.601 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,00f646d78c957baf,e759e5a339c2c008] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=00f646d78c957baf, duration=5 ms] 2025-04-08 21:05:12.621 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:05:12.621 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:05:12.621 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:05:13.195 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,27f73138ba2d3c7b,e62e923095b6e17d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=27f73138ba2d3c7b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:05:13.195 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0f27f2b880d2b61f,4ad5fa361d513a87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=0f27f2b880d2b61f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:05:13.195 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0f27f2b880d2b61f,4ad5fa361d513a87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=0f27f2b880d2b61f, duration=4 ms] 2025-04-08 21:05:13.195 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,a0e874f469fcbb71,e576137f18d42952] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=a0e874f469fcbb71, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:05:13.196 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,a0e874f469fcbb71,e576137f18d42952] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=a0e874f469fcbb71, duration=5 ms] 2025-04-08 21:05:13.196 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,456a794bd39dbe24,8015405784c53fa9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=456a794bd39dbe24, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:05:13.198 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,27f73138ba2d3c7b,e62e923095b6e17d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=27f73138ba2d3c7b, duration=7 ms] 2025-04-08 21:05:13.208 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:05:13.208 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:05:13.208 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:05:13.287 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,456a794bd39dbe24,8015405784c53fa9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=456a794bd39dbe24, duration=96 ms] 2025-04-08 21:05:13.340 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,2d60ce41da660853,cc7518485f97a7a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=2d60ce41da660853, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:05:13.344 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,2d60ce41da660853,cc7518485f97a7a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=2d60ce41da660853, duration=23 ms] 2025-04-08 21:05:14.139 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:14.139 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:14.552 WARN 26087 --- [ocp-async-5,a93949ec2ddcfcb5,e42e8f6a851622e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:14.552 INFO 26087 --- [ocp-async-7,965260dbbf83a40f,c07775e1e25af37b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:15.185 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,242a87c28c22c61a,24f142b65ba7c843] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=242a87c28c22c61a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:05:15.185 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,776b2fe165005788,3370891d9122413d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=776b2fe165005788, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:05:15.185 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,776b2fe165005788,3370891d9122413d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=776b2fe165005788, duration=3 ms] 2025-04-08 21:05:15.185 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,242a87c28c22c61a,24f142b65ba7c843] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=242a87c28c22c61a, duration=3 ms] 2025-04-08 21:05:15.186 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,875b8dd448c2c243,214e0a0d398ae669] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=875b8dd448c2c243, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:05:15.186 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,875b8dd448c2c243,214e0a0d398ae669] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=875b8dd448c2c243, duration=2 ms] 2025-04-08 21:05:15.195 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:05:15.195 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:05:15.195 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:05:15.355 INFO 26087 --- [prometheus-minute-schedule-1,cf758c84a5dad08a,76f88fc15f0512a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:15.381 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,59a83dca4d77f24b,1ff2d749f35f5cd7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=59a83dca4d77f24b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:05:15.381 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,59a83dca4d77f24b,1ff2d749f35f5cd7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=59a83dca4d77f24b, duration=4 ms] 2025-04-08 21:05:15.578 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,f564f03da1bd43fb,5431540533db334e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=f564f03da1bd43fb, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:05:15.581 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,f564f03da1bd43fb,5431540533db334e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=f564f03da1bd43fb, duration=6 ms] 2025-04-08 21:05:16.510 INFO 26087 --- [ocp-schedules-5,b98e976c980d472b,06fe77897ebe073f] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:05:16.510+08:00, tenantSize=0 2025-04-08 21:05:16.517 INFO 26087 --- [ocp-schedules-8,3eddb855e09b5f3a,02cae3ece8ce2317] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:05:16.517 INFO 26087 --- [ocp-schedules-23,f7115b850cd0985a,f0a7df9ab5a9b762] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:05:16.521 INFO 26087 --- [ocp-schedules-22,2954e554ff41a239,bba91b69e9f19260] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:05:16.523 INFO 26087 --- [ocp-schedules-8,3eddb855e09b5f3a,02cae3ece8ce2317] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:05:16.524 INFO 26087 --- [ocp-schedules-23,f7115b850cd0985a,f0a7df9ab5a9b762] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:05:16.524 INFO 26087 --- [ocp-schedules-2,2f507bafd3749c37,731559325a07f9b7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:05:16.524 INFO 26087 --- [ocp-schedules-29,59d2bcf4b81b98f5,a3ef33ce7abc1dd5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:05:16.529 INFO 26087 --- [ocp-schedules-25,8c5c066433159d8b,e9961404edd6bc33] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:05:16.534 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:05:16.535 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:05:16.538 INFO 26087 --- [ocp-schedules-15,b850ce34f9450008,bbeb315c7dde143e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:05:16.538 INFO 26087 --- [ocp-async-6,1059c40b34087390,6fa7fd7d658c5740] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:05:16.538 INFO 26087 --- [ocp-schedules-15,b850ce34f9450008,bbeb315c7dde143e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:05:16.539 INFO 26087 --- [ocp-schedules-3,518a656c94a202ea,6d5174b2890b3f1f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:05:16.540 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:05:16.540 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:05:16.540 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:05:16.540 INFO 26087 --- [ocp-schedules-3,518a656c94a202ea,6d5174b2890b3f1f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:05:16.541 INFO 26087 --- [ocp-schedules-21,23ffffe2f2b96d52,27461d8b9a012817] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:05:16.541 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:05:16.541 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:05:16.541 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:05:16.542 INFO 26087 --- [ocp-schedules-21,23ffffe2f2b96d52,27461d8b9a012817] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:05:16.544 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:05:16.544 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:05:16.544 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:05:16.546 INFO 26087 --- [ocp-schedules-27,af40d6a954282b84,aeb3d1594475820b] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:05:16.551 INFO 26087 --- [ocp-schedules-10,415a25c114b57875,4b7b4ceb8a0424cb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:05:16.551 INFO 26087 --- [ocp-schedules-10,415a25c114b57875,4b7b4ceb8a0424cb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:05:16.552 WARN 26087 --- [ocp-async-3,74ddb449d5fc7db7,2600022ac1570846] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:16.552 INFO 26087 --- [ocp-async-10,383cefe8d6bf49c2,01c4040799a9c075] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:16.555 INFO 26087 --- [ocp-schedules-11,defa84d6329b179b,5a38348d7909c488] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 21:05:16.555 INFO 26087 --- [ocp-schedules-11,defa84d6329b179b,5a38348d7909c488] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 21:05:16.567 INFO 26087 --- [ocp-schedules-29,389dc7f7b6c9a045,2a4387306aa6dbc5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:05:16.567 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:05:16.567 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:05:16.568 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.568 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:05:16.569 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.569 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.570 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.570 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:05:16.572 INFO 26087 --- [ocp-schedules-24,c9dfa0ff0a749a5d,70ea132af01982ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:05:16.572 INFO 26087 --- [ocp-schedules-24,9eeaea26bd47b105,d6a25d34f0d50d98] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:05:16.573 INFO 26087 --- [ocp-schedules-2,4c669261c988ce74,fc73e35f86e18eac] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:05:16.574 INFO 26087 --- [ocp-schedules-2,ff5375d4eb4ead71,67ae8f92da2b279a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:05:16.574+08:00, clusters=[] 2025-04-08 21:05:16.575 INFO 26087 --- [ocp-schedules-1,728032a4828de8c5,0be1abf5a4718398] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:05:16.575 INFO 26087 --- [ocp-schedules-16,8d503e120a012e4a,42ec86cefb3d0c07] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:05:16.577 INFO 26087 --- [ocp-schedules-18,6c9c4d9c68203432,821febaf185b4715] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:05:16.577+08:00, clusters=[] 2025-04-08 21:05:16.582 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.582 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.582 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.583 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:05:16.584 INFO 26087 --- [ocp-schedules-19,c5bf690db1a1273e,b9780fb9660d51f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":356,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":1} 2025-04-08 21:05:16.584 INFO 26087 --- [ocp-schedules-19,c5bf690db1a1273e,b9780fb9660d51f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:05:16.591 INFO 26087 --- [ocp-schedules-3,eb5f600372289509,55cdfeab5f4dd3c4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:05:16.591 INFO 26087 --- [ocp-schedules-3,eb5f600372289509,55cdfeab5f4dd3c4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:05:16.592 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.592 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.593 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.593 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:05:16.603 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.604 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.604 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.604 INFO 26087 --- [ocp-schedules-27,2ee245b3a8cc6a4f,2718650a2881250c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:05:16.605 INFO 26087 --- [ocp-schedules-27,2ee245b3a8cc6a4f,2718650a2881250c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:05:16.605 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:05:16.606 INFO 26087 --- [ocp-schedules-27,2ee245b3a8cc6a4f,2718650a2881250c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:05:16.606 INFO 26087 --- [ocp-schedules-27,2ee245b3a8cc6a4f,2718650a2881250c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:05:16.607 INFO 26087 --- [ocp-schedules-21,b67e8b64d1621c4b,ec0d2b70a61f3e33] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:05:16.609 INFO 26087 --- [ocp-schedules-6,3fb0ae7e650ae0c8,2967b54157ba67cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:05:16.615 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.615 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.616 INFO 26087 --- [ocp-schedules-4,d0c2a8b3d3238d05,758dff72be043562] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:05:16.616 INFO 26087 --- [ocp-schedules-4,d0c2a8b3d3238d05,758dff72be043562] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:05:16.616 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.616 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:05:16.622 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.622 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.623 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.623 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:05:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:05:16.624185+08:00, lastProcessTime=2025-04-08T21:05:06.624893+08:00 2025-04-08 21:05:16.627 INFO 26087 --- [ocp-schedules-9,7f3c66d0c1ee0bfa,cc842a873a8ed463] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:05:16.627 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.628 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.628 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.628 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:05:16.632 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.632 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.633 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.633 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:05:16.638 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.638 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.638 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.639 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:05:16.643 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.643 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.644 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.644 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:05:16.647 INFO 26087 --- [ocp-schedules-28,ac108a088d10d6ea,4c56baba77042b3c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:05:16.648 INFO 26087 --- [ocp-schedules-28,ac108a088d10d6ea,4c56baba77042b3c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:05:16.648 INFO 26087 --- [ocp-schedules-28,ac108a088d10d6ea,4c56baba77042b3c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:05:16.648 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.649 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.649 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.649 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:05:16.656 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.656 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.657 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.658 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:05:16.662 INFO 26087 --- [ocp-schedules-20,3cfeaa115586cf34,524fb698a8aea2bc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:05:16.668 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.669 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.669 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.669 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:05:16.673 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.673 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.674 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.674 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:05:16.675 INFO 26087 --- [ocp-schedules-20,3cfeaa115586cf34,524fb698a8aea2bc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:05:16.675 INFO 26087 --- [ocp-schedules-20,3cfeaa115586cf34,524fb698a8aea2bc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:05:16.678 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.678 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.679 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.679 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:05:16.682 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.683 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.683 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.684 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:05:16.687 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.687 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.689 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.689 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:05:16.693 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.694 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.694 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.694 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:05:16.697 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.698 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.698 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.698 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:05:16.702 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.702 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.703 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.703 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:05:16.706 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.706 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.707 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.707 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:05:16.711 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.711 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.711 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.711 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:05:16.715 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.715 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.715 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.715 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:05:16.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:05:16.624185+08:00 2025-04-08 21:05:16.719 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.719 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.719 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.720 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:05:16.723 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.723 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.724 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.724 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:05:16.727 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.727 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.728 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.728 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:05:16.731 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.732 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.732 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.733 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:05:16.736 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.736 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.737 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.737 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:05:16.740 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.740 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.741 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.741 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:05:16.744 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.744 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.745 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.745 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:05:16.748 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.748 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.749 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.749 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:05:16.752 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.753 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.753 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.754 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:05:16.757 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.757 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.757 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.758 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:05:16.761 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.761 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.761 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.761 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:05:16.767 INFO 26087 --- [ocp-schedules-14,42e6eb89358c2afc,ef0c4e850dc5c07c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:05:16.767 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.767 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.768 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.768 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:05:16.772 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.772 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.772 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.773 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:05:16.779 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.779 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.780 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.780 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:05:16.785 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.785 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.785 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.786 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:05:16.790 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.790 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.791 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.791 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:05:16.792 INFO 26087 --- [ocp-schedules-23,d4a29fdc10d191c9,8b3fa02222f47c86] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:05:16.792 INFO 26087 --- [ocp-schedules-23,d4a29fdc10d191c9,8b3fa02222f47c86] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:05:16.793 INFO 26087 --- [ocp-schedules-23,d4a29fdc10d191c9,8b3fa02222f47c86] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:05:16.793 INFO 26087 --- [ocp-schedules-23,d4a29fdc10d191c9,8b3fa02222f47c86] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:05:16.793 INFO 26087 --- [ocp-schedules-23,d4a29fdc10d191c9,8b3fa02222f47c86] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:05:16.793 INFO 26087 --- [ocp-schedules-23,d4a29fdc10d191c9,8b3fa02222f47c86] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:05:16.794 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.794 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.795 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.795 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:05:16.798 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.798 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.799 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.799 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:05:16.802 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.802 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.802 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.802 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:05:16.810 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.810 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.811 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.811 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:05:16.823 INFO 26087 --- [ocp-schedules-12,dc9bd29c849fc787,2f8a4bf06ed3748c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:05:16.823 INFO 26087 --- [ocp-schedules-26,2f181196a7adb07f,b088466dc4687336] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61, current targets ,,,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61 2025-04-08 21:05:16.824 INFO 26087 --- [ocp-schedules-12,dc9bd29c849fc787,2f8a4bf06ed3748c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:05:16.827 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.827 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.828 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.828 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:05:16.833 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.834 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.834 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.834 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:05:16.839 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.840 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.840 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.840 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:05:16.845 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.845 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.845 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.846 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:05:16.849 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.849 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.849 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.850 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:05:16.853 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.853 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.853 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.853 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:05:16.857 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:05:16.857 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:05:16.857 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.857 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:05:16.858 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.858 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.859 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.859 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:05:16.863 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.863 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:05:16.863 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:16.865 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:05:16.868 INFO 26087 --- [ocp-schedules-17,ee2347a92dcab204,d0ccfc92331fd35c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:05:16.876 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:05:16.877 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:05:16.878 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:05:16.879 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:05:16.880 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:05:16.881 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:05:16.882 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:05:16.883 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:05:16.884 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:05:16.885 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:05:16.886 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 21:05:16.887 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:05:16.888 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:05:16.889 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:05:16.890 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:05:16.891 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:05:16.892 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:05:16.893 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:05:16.894 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:05:16.895 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:05:16.896 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:05:16.897 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:05:16.898 INFO 26087 --- [ocp-schedules-11,d086376b04159dc4,38deee12a24aedd8] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:05:17.139 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:17.140 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:17.223 INFO 26087 --- [ocp-schedules-16,0766b58d46efe3c1,5308821d234e07e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:05:18.183 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,9ea0caaacbc8abf8,c3aa1466b6b4d6cb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=9ea0caaacbc8abf8, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:05:18.184 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,9ea0caaacbc8abf8,c3aa1466b6b4d6cb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=9ea0caaacbc8abf8, duration=5 ms] 2025-04-08 21:05:18.355 INFO 26087 --- [prometheus-minute-schedule-1,08116c815bf11131,9e07dfe5c65b007a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:18.552 WARN 26087 --- [ocp-async-2,1b2e9f1ceac8c329,d35b3fa1abab25cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:18.552 INFO 26087 --- [ocp-async-9,333b6e30d083aa60,6742cbcdb872d880] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:20.140 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:20.141 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:20.552 WARN 26087 --- [ocp-async-1,7fff8415b9b6b309,f634168d01b98281] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:20.552 INFO 26087 --- [ocp-async-9,a1d0773972a0add1,51c6d41bc5af0f34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:05:21.355 INFO 26087 --- [prometheus-minute-schedule-1,9679f52d72dde671,57149cf03d44af87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:22.552 WARN 26087 --- [ocp-async-8,997a96e3b398b354,13e6e608e466821e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:22.552 INFO 26087 --- [ocp-async-3,fbf0ede79a6e6fac,0830ac6132f0a91a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:23.141 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:23.142 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:24.355 INFO 26087 --- [prometheus-minute-schedule-1,1fb802c95fd93ca4,70bc49eebdaab268] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:24.552 WARN 26087 --- [ocp-async-2,50d76bbb77354eb2,aae0071beadc0cac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:24.553 INFO 26087 --- [ocp-async-10,77ddb591c051e0a7,596bd96b456f4b17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:26.142 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:26.143 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:26.387 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOverPreCheck, client 10.11.12.102, traceId b7886c493b3b3c08] 2025-04-08 21:05:26.430 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOverPreCheck?null, client=10.11.12.102, traceId=b7886c493b3b3c08, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(TakeOverClusterPreCheckParam), args=TakeOverClusterPreCheckParam(connectionMode=direct, clusterName=null, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=192.168.1.61, port=2881),] 2025-04-08 21:05:26.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=192.168.1.61, port=2881, database=oceanbase) 2025-04-08 21:05:26.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:05:26.446 INFO 26087 --- [obsdk-init-thread-4,b7886c493b3b3c08,c3482b1c1ab99312] com.alibaba.druid.pool.DruidDataSource : {dataSource-6,ob-connector-datasource_direct_sys_root_192.168.1.61_2881} inited 2025-04-08 21:05:26.446 INFO 26087 --- [obsdk-init-thread-4,b7886c493b3b3c08,c3482b1c1ab99312] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.61, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:05:26.446 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:05:26.448 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.449 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.449 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.450 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.450 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.451 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.451 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.452 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.453 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.454 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.454 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.455 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 21:05:26.461 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.462 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.462 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.463 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 21:05:26.468 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.468 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.469 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.469 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 21:05:26.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.475 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.475 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.476 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.481 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.481 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 21:05:26.484 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.485 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.485 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.486 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.487 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.487 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 21:05:26.493 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.493 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.498 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.498 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 21:05:26.520 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.521 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.526 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.529 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 21:05:26.534 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.535 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.536 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.536 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.538 INFO 26087 --- [ocp-async-7,b22557fc9dd14a0c,3701ab906fe4494c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:05:26.541 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:05:26.542 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.544 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:05:26.553 WARN 26087 --- [ocp-async-4,c4e06e408112444d,3b7c8bc71b387ea8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:26.553 INFO 26087 --- [ocp-async-4,9de77db47ecaa3f3,c4e64ea33fec2ec6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:26.568 INFO 26087 --- [ocp-schedules-29,c1ba983a8f39bb79,ef8e9c6dd8b40701] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:05:26.574 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.575 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.579 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.579 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:05:26.603 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.603 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.604 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.604 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.605 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.606 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.606 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.607 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.607 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.608 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:05:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:05:26.623275+08:00, lastProcessTime=2025-04-08T21:05:16.624185+08:00 2025-04-08 21:05:26.626 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.626 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.631 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.631 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:05:26.637 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.637 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.639 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.640 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:05:26.668 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.668 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.669 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.669 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.671 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.671 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.673 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.673 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:05:26.688 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.688 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.689 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.690 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:05:26.692 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.692 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.693 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.693 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:05:26.712 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.712 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.713 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.713 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.714 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.715 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.715 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.716 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.716 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.717 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:05:26.732 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.732 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.733 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:05:26.623275+08:00 2025-04-08 21:05:26.734 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.734 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:05:26.753 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.754 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.755 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.755 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.756 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.756 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.757 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.757 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 21:05:26.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.801 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.801 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.802 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.802 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:05:26.803 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.803 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:05:26.804 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.804 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:05:26.811 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.811 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.812 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.812 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.813 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.813 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.814 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.814 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:05:26.815 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:05:26.815 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:05:26.828 INFO 26087 --- [ocp-schedules-6,87edce2e16a5cb30,81411ddcc2af04c2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:05:26.830 INFO 26087 --- [ocp-schedules-6,87edce2e16a5cb30,81411ddcc2af04c2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:05:26.849 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 21:05:26.857 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b7886c493b3b3c08,779357a6556f7eeb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.11.12.102, traceId=b7886c493b3b3c08, duration=473 ms] 2025-04-08 21:05:26.982 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,1d803efc71ed798a,5c0fe4991c2a480d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=1d803efc71ed798a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 21:05:26.982 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,72da9c26cca4fcdf,328d7fd6e18007a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=72da9c26cca4fcdf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 21:05:27.000 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,1d803efc71ed798a,5c0fe4991c2a480d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=1d803efc71ed798a, duration=22 ms] 2025-04-08 21:05:27.033 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,72da9c26cca4fcdf,328d7fd6e18007a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=72da9c26cca4fcdf, duration=55 ms] 2025-04-08 21:05:27.266 INFO 26087 --- [ocp-schedules-1,7b182c8113d54fd7,c05dd2e9bd04d68e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:05:27.355 INFO 26087 --- [prometheus-minute-schedule-1,2148f41a0c98fc40,1702bb427d580bed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:28.553 WARN 26087 --- [ocp-async-6,759f92220c3f1250,557f8d7d4360c5b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:28.553 INFO 26087 --- [ocp-async-3,5a640899b32c4d5f,494b0758a22f43af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:29.143 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:29.144 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:30.355 INFO 26087 --- [prometheus-minute-schedule-1,bd76a03844cb6455,698397e6ae4af4be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:30.553 WARN 26087 --- [ocp-async-3,695173f4fcdaab8f,2f9d1ec2661eea58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:30.553 INFO 26087 --- [ocp-async-10,30bbbc7dc821a175,a64214f24b9bddfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:32.144 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:32.145 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:32.553 WARN 26087 --- [ocp-async-8,c362622540f81380,7aaf8431c1b912e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:32.553 INFO 26087 --- [ocp-async-9,828c1e081a8ae516,d94f57bc088d9d0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:33.355 INFO 26087 --- [prometheus-minute-schedule-1,fafe70b763ca836d,42196e541ff78ccb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:34.317 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,d43ac38b9f40b92f,3483493243513b69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=d43ac38b9f40b92f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 21:05:34.319 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,7d3c4139fec88169,380352e7262bfb04] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.11.12.102, traceId=7d3c4139fec88169, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 21:05:34.321 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,d43ac38b9f40b92f,3483493243513b69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=d43ac38b9f40b92f, duration=8 ms] 2025-04-08 21:05:34.323 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-2,7d3c4139fec88169,380352e7262bfb04] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2025-04-08 21:05:34.343 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,7d3c4139fec88169,380352e7262bfb04] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.11.12.102, traceId=7d3c4139fec88169, duration=30 ms] 2025-04-08 21:05:34.553 WARN 26087 --- [ocp-async-6,251b8b5d9c7dbee9,9201e7682403f3b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:34.553 INFO 26087 --- [ocp-async-9,7ab3325b01595df6,032a5473227d486e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:34.978 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:05:34.979 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:05:35.145 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:35.145 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:36.355 INFO 26087 --- [prometheus-minute-schedule-1,ee8df019591c4716,9bff40194364c7f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:36.539 INFO 26087 --- [ocp-async-9,d2c0ceec97062d60,f48c2de611902a12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:05:36.554 WARN 26087 --- [ocp-async-3,5ca8325bee1ed425,71be364c1044be90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:36.554 INFO 26087 --- [ocp-async-3,e8b39aa42d2c329b,e50bf22d46fbb9d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:36.555 INFO 26087 --- [ocp-schedules-24,c9f8afe667216323,f12d2e7d4d80f454] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:05:36.569 INFO 26087 --- [ocp-schedules-30,b4302c51fed412e6,3390e9d2601d3979] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:05:36.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:05:36.624485+08:00, lastProcessTime=2025-04-08T21:05:26.623275+08:00 2025-04-08 21:05:36.676 INFO 26087 --- [ocp-schedules-8,59b0b4e98bb45648,5f137507286a408a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:05:36.682 INFO 26087 --- [ocp-schedules-8,59b0b4e98bb45648,5f137507286a408a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:05:36.682 INFO 26087 --- [ocp-schedules-8,59b0b4e98bb45648,5f137507286a408a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:05:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:05:36.624485+08:00 2025-04-08 21:05:36.838 INFO 26087 --- [ocp-schedules-2,eaaf05b7a8d0442f,55f00bebb690a1c5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:05:36.840 INFO 26087 --- [ocp-schedules-2,eaaf05b7a8d0442f,55f00bebb690a1c5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:05:37.290 INFO 26087 --- [ocp-schedules-6,1a1554d3cfa952f6,bf69980e6eb08eb6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:05:38.146 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:38.146 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:38.554 WARN 26087 --- [ocp-async-3,41588dde3ff7dd4e,9438deb3a1ed38db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:38.554 INFO 26087 --- [ocp-async-10,0f8138f763739882,736955d8c9c7892c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:39.355 INFO 26087 --- [prometheus-minute-schedule-1,2787fb351e3b7614,d49816add507278c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:40.001 INFO 26087 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:05:40.001 INFO 26087 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:05:40.554 WARN 26087 --- [ocp-async-2,e9fbd31eaf2bfd97,bd34815bf41ea96b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:40.554 INFO 26087 --- [ocp-async-4,941eb54ebd9970a5,2d1f7729805e4756] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:41.146 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:41.147 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:42.355 INFO 26087 --- [prometheus-minute-schedule-1,f9f33b9c648bd7b6,93648f0b3f062512] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:42.554 WARN 26087 --- [ocp-async-4,bce7f24d436d7e99,78fef2dcf49f2ca4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:42.554 INFO 26087 --- [ocp-async-1,fecbf7fd3a1c1bee,ddb0de487e980844] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:44.147 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:44.148 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:44.554 WARN 26087 --- [ocp-async-6,3b968478e1625116,e5724345e3a9ac0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:44.554 INFO 26087 --- [ocp-async-8,d355f0cd2d64701c,750facd770de5863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:45.355 INFO 26087 --- [prometheus-minute-schedule-1,4c7c19a264fbf977,0a5cae8167a87f93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:46.521 INFO 26087 --- [ocp-schedules-30,ee38ca03a9a6ad02,fb897c9405778a38] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:05:46.522 INFO 26087 --- [ocp-schedules-29,dda39794e5bdd91a,5772c452c508a136] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:05:46.522 INFO 26087 --- [ocp-schedules-29,dda39794e5bdd91a,5772c452c508a136] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-4,58c0bfad0b900954,2a4f806d7a2f5361] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-4,58c0bfad0b900954,2a4f806d7a2f5361] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-24,29eaec826642f073,8a1177ce91a786e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-24,29eaec826642f073,8a1177ce91a786e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-21,61d230e339feb1db,0c9d5ad093b64ebe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-13,5a5fd34ee1145dde,faa2f6d4e9a09b3f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:05:46.524 INFO 26087 --- [ocp-schedules-11,f9d68a13f3d1b5e8,cc43d4078d928975] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:05:46.531 INFO 26087 --- [ocp-schedules-27,92d0f0992e51d36d,a5e8849f00242db5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:05:46.540 INFO 26087 --- [ocp-async-7,febcbd5f577dec5a,a8824890f90b8090] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:05:46.554 WARN 26087 --- [ocp-async-5,292f963e3ebea668,d1041616b747d72f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:46.555 INFO 26087 --- [ocp-async-10,9c96f92f5e53208e,60c8b5591ef3f602] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:46.571 INFO 26087 --- [ocp-schedules-12,f189b402222f0d78,79be19b6116d2ae4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:05:46.576 INFO 26087 --- [ocp-schedules-16,109ab70c237eeba7,53dc3af388dc8bc6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:05:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:05:46.623261+08:00, lastProcessTime=2025-04-08T21:05:36.624485+08:00 2025-04-08 21:05:46.735 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:05:46.623261+08:00 2025-04-08 21:05:46.889 INFO 26087 --- [ocp-schedules-25,74ce2bf4bd7b0c8e,f3d7b7380460e0a1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:05:46.891 INFO 26087 --- [ocp-schedules-25,74ce2bf4bd7b0c8e,f3d7b7380460e0a1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:05:47.148 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:47.149 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:47.367 INFO 26087 --- [ocp-schedules-24,e7ebb81b6992f7cb,2728f2e4760636d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:05:48.355 INFO 26087 --- [prometheus-minute-schedule-1,7dfe4fccfa6a6a06,cc029444508a3030] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:48.555 WARN 26087 --- [ocp-async-1,8648e3609f94ece2,6e94829f7fff14a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:48.555 INFO 26087 --- [ocp-async-9,f633ae3fbec377e1,d8529b14bfa76431] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:50.149 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:50.151 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:50.555 WARN 26087 --- [ocp-async-6,545617a6d505076c,bb4d18f69d9ba801] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:50.555 INFO 26087 --- [ocp-async-3,6a2490e7b22f560a,9b4360b71b0ae4a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:51.355 INFO 26087 --- [prometheus-minute-schedule-1,cc04ff431b678e93,47aa3e51ce5ddc8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:52.555 WARN 26087 --- [ocp-async-5,0e8989debd5b3273,6f721a9a31a4a714] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:52.555 INFO 26087 --- [ocp-async-4,df093d249411e638,b54acb3e510b2e72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:53.151 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:53.152 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:53.626 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,fb194d00d53a9de4,44b9059af51f872a] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=fb194d00d53a9de4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744117553621288,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 21:05:53.644 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,fb194d00d53a9de4,44b9059af51f872a] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=fb194d00d53a9de4, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744117553621288,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=22 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 21:05:54.355 INFO 26087 --- [prometheus-minute-schedule-1,3565f1cc5d21db2a,40b0bc4fefa407ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:54.555 WARN 26087 --- [ocp-async-4,f64efce4136e7bf9,c67f6f6eb99ecb48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:54.555 INFO 26087 --- [ocp-async-1,1785d1539faa32a2,76d545b4c2db24d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:56.152 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:56.153 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:05:56.540 INFO 26087 --- [ocp-async-9,dc6375186dd0e0c7,603911645e73290c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:05:56.555 WARN 26087 --- [ocp-async-6,368127db5fa3e082,eca7c87003d3fb99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:56.555 INFO 26087 --- [ocp-async-3,6ca603fc256756f6,25f38f30c3f9e8f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:56.573 INFO 26087 --- [ocp-schedules-1,29baa98e609cc68a,41d994a7e0dd2e44] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:05:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:05:56.623516+08:00, lastProcessTime=2025-04-08T21:05:46.623261+08:00 2025-04-08 21:05:56.683 INFO 26087 --- [ocp-schedules-16,42d60cf689f26b30,8dbf0e9b4c01cdd2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:05:56.740 INFO 26087 --- [ocp-schedules-16,42d60cf689f26b30,8dbf0e9b4c01cdd2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:05:56.740 INFO 26087 --- [ocp-schedules-16,42d60cf689f26b30,8dbf0e9b4c01cdd2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:05:56.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:05:56.623516+08:00 2025-04-08 21:05:56.894 INFO 26087 --- [ocp-schedules-6,2615be8d57568449,e3ae704912ec954e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:05:56.895 INFO 26087 --- [ocp-schedules-6,2615be8d57568449,e3ae704912ec954e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:05:57.355 INFO 26087 --- [prometheus-minute-schedule-1,2fc13cc0f9c7f027,9ae4b00a6a25bf27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117560, interval=60 2025-04-08 21:05:57.399 INFO 26087 --- [ocp-schedules-20,653e0967339d70f4,3d70473eccd27ad9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:05:58.555 WARN 26087 --- [ocp-async-2,4e86e656fe2f5c33,3a7b16f26ce99f9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:05:58.555 INFO 26087 --- [ocp-async-7,317da219cda2b438,8c5d6c44c059a0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:05:59.153 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:05:59.154 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:00.355 INFO 26087 --- [prometheus-minute-schedule-1,0d51d2ab8e787311,22d45f0671c0a6b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:00.555 WARN 26087 --- [ocp-async-1,8825971383fcaf01,cc59c98ad89a9d4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:00.556 INFO 26087 --- [ocp-async-10,8709688afd5cd8ee,c55a5f563dd321ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:02.154 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:02.155 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:02.556 WARN 26087 --- [ocp-async-6,16cc1fffe08bde74,5fd7ad5562add745] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:02.556 INFO 26087 --- [ocp-async-9,4e31ba2a2042ff13,10063d80b833df36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:03.355 INFO 26087 --- [prometheus-minute-schedule-1,fa71fcb7369d08f9,70cd701d4e5fcb3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:04.556 WARN 26087 --- [ocp-async-2,708626b650ff4775,342bd4d60b35754a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:04.556 INFO 26087 --- [ocp-async-3,97f7c9dc67c7476f,35c5cbdd33a936f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:05.155 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:05.156 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:06.217 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:02:06.167, end=2025-04-08T21:04:06.208} 2025-04-08 21:06:06.220 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:06:06.220 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:06:06.355 INFO 26087 --- [prometheus-minute-schedule-1,6078cf03f93e5468,a4db5820e0aa5f78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:06.355 INFO 26087 --- [prometheus-cache-monitor1,d6dc824353b793d0,766d9c8838b91afa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":0,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":364,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":1} 2025-04-08 21:06:06.355 INFO 26087 --- [prometheus-cache-monitor1,d6dc824353b793d0,766d9c8838b91afa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:06:06.540 INFO 26087 --- [ocp-async-3,cf12d97401648087,11b7c95634045efc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:06:06.556 WARN 26087 --- [ocp-async-1,f7ca50314b8511f0,3db6d94f90de1245] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:06.556 INFO 26087 --- [ocp-async-4,54b4ef83749a4af6,74d42e560e096901] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:06.574 INFO 26087 --- [ocp-schedules-10,4b40dd1069fdfd8f,0704b33b2bd29cdc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:06:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:06:06.623849+08:00, lastProcessTime=2025-04-08T21:05:56.623516+08:00 2025-04-08 21:06:06.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:06:06.623849+08:00 2025-04-08 21:06:06.901 INFO 26087 --- [ocp-schedules-26,3b7a4d6ea7521e01,8eb59b22dd56f0b8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:06:06.903 INFO 26087 --- [ocp-schedules-26,3b7a4d6ea7521e01,8eb59b22dd56f0b8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:06:07.456 INFO 26087 --- [ocp-schedules-24,f8dfc0bb4051187b,7346c3f2b91aad26] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:06:08.156 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:08.156 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:08.556 WARN 26087 --- [ocp-async-8,2a7c23c412c60b83,d5fe46f9f058994d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:08.556 INFO 26087 --- [ocp-async-6,9d16cd0fc0853738,be84db247f7b4036] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:09.355 INFO 26087 --- [prometheus-minute-schedule-1,9b65e88df34b8a54,9fce15b0fe191524] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:10.002 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:06:10.002 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:06:10.556 WARN 26087 --- [ocp-async-7,ab3b9f726b6e84e1,f866e15e6387cf55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:10.556 INFO 26087 --- [ocp-async-5,f3ab2c05a56138ad,e4c6b492b1b546d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:06:11.157 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:11.157 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:12.330 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=1, affectRows=1 2025-04-08 21:06:12.330 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=1, validCount=1 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=26, affectRows=26 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=24, affectRows=24 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=12, affectRows=12 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=42, affectRows=42 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=1, affectRows=1 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=9, affectRows=9 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=41, affectRows=41 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=35, affectRows=35 2025-04-08 21:06:12.351 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=17, affectRows=17 2025-04-08 21:06:12.352 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=1, validCount=1 2025-04-08 21:06:12.352 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=9, validCount=9 2025-04-08 21:06:12.352 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=24, validCount=24 2025-04-08 21:06:12.352 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=12, validCount=12 2025-04-08 21:06:12.353 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=26, validCount=26 2025-04-08 21:06:12.353 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=42, validCount=42 2025-04-08 21:06:12.353 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=17, validCount=17 2025-04-08 21:06:12.353 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=35, validCount=35 2025-04-08 21:06:12.353 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=41, validCount=41 2025-04-08 21:06:12.355 INFO 26087 --- [prometheus-minute-schedule-1,5ad34663f1ce1af3,a8fccd6bbe6230ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:12.357 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=38, affectRows=38 2025-04-08 21:06:12.357 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=31, affectRows=31 2025-04-08 21:06:12.357 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=30, affectRows=30 2025-04-08 21:06:12.357 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=29, affectRows=29 2025-04-08 21:06:12.357 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117500, nodeCount=28, affectRows=28 2025-04-08 21:06:12.358 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=30, validCount=30 2025-04-08 21:06:12.358 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=38, validCount=38 2025-04-08 21:06:12.358 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=28, validCount=28 2025-04-08 21:06:12.358 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=29, validCount=29 2025-04-08 21:06:12.358 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117500, nodeCount=30, validCount=30 2025-04-08 21:06:12.556 WARN 26087 --- [ocp-async-1,bb330237f6ade31a,4df1010e148cdc4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:12.556 INFO 26087 --- [ocp-async-3,c54bf2cc5cc7cca5,ec37840733d9d310] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:14.157 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:14.158 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:14.557 WARN 26087 --- [ocp-async-3,ca4c5c6473183e2e,fb8c0f565724709e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:14.557 INFO 26087 --- [ocp-async-4,e03df5194b20da75,5f8605e72b7dd6c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:15.355 INFO 26087 --- [prometheus-minute-schedule-1,b97e8f3d02793189,6ae0dcde5e6716dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:16.517 INFO 26087 --- [ocp-schedules-6,186de1605b357f93,001658cf5efdc56f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:06:16.518 INFO 26087 --- [ocp-schedules-5,30f1cbc3de6083c3,54c3abfc45cf3ade] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:06:16.521 INFO 26087 --- [ocp-schedules-3,063781398248b118,d8a03b61b95c5d48] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:06:16.521 INFO 26087 --- [ocp-schedules-8,f109a7f10b4f9e21,2451950992ccaaeb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:06:16.521 INFO 26087 --- [ocp-schedules-8,f109a7f10b4f9e21,2451950992ccaaeb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-27,1471850a5308c9ff,032eacf5ccb66b54] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-27,1471850a5308c9ff,032eacf5ccb66b54] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-10,d9c84931d886fd51,18e2f5e4dd701f6c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-10,d9c84931d886fd51,18e2f5e4dd701f6c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-26,177d84584bc96f79,f5d580a2d4a58316] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-13,186b789a1efe6085,0a5ea4f400691044] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:06:16.524 INFO 26087 --- [ocp-schedules-11,8f3a28ca7c074990,650341b06a0a4b19] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:06:16.526 INFO 26087 --- [ocp-schedules-5,30f1cbc3de6083c3,54c3abfc45cf3ade] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:06:16.535 INFO 26087 --- [ocp-schedules-6,186de1605b357f93,001658cf5efdc56f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:06:16.538 INFO 26087 --- [ocp-schedules-12,1d3cd14960016f39,d7802eb7d385c21b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:06:16.540 INFO 26087 --- [ocp-async-7,447e6e2906438b16,ab2cc8470a5ca692] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:06:16.541 INFO 26087 --- [ocp-schedules-9,5038295b8b00b82d,44ccd5c82546fdc4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:06:16.541 INFO 26087 --- [ocp-schedules-9,5038295b8b00b82d,44ccd5c82546fdc4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:06:16.557 WARN 26087 --- [ocp-async-10,ce415582451f7ded,c5fcf0ceab4f5802] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:16.557 INFO 26087 --- [ocp-async-2,9d45c149e0c33353,a1eba5f2478a11fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:16.575 INFO 26087 --- [ocp-schedules-2,7e91433311d20e8e,14d3a4191c30c380] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:06:16.575 INFO 26087 --- [ocp-schedules-30,8c7f6c31d5c811cd,e212d1b85f29a6c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:06:16.575 INFO 26087 --- [ocp-schedules-8,356acdae9834ee44,97060d8de3777d27] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:06:16.575+08:00, clusters=[] 2025-04-08 21:06:16.577 INFO 26087 --- [ocp-schedules-27,9d8f91cc6412589d,e4bd47583d8c4802] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:06:16.578 INFO 26087 --- [ocp-schedules-3,13e2c4c9cbb2fdec,4d2ea536ef8a340b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:06:16.578 INFO 26087 --- [ocp-schedules-10,250fbf98a8c756ad,cc26fee32c4bf2b3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:06:16.578+08:00, clusters=[] 2025-04-08 21:06:16.593 INFO 26087 --- [ocp-schedules-26,9d4fdec3666ca357,e2483ae62af8aba6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:06:16.593 INFO 26087 --- [ocp-schedules-26,9d4fdec3666ca357,e2483ae62af8aba6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:06:16.606 INFO 26087 --- [ocp-schedules-11,abd9fd7c71e79c5c,8893de11e0d23999] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:06:16.606 INFO 26087 --- [ocp-schedules-11,abd9fd7c71e79c5c,8893de11e0d23999] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:06:16.607 INFO 26087 --- [ocp-schedules-11,abd9fd7c71e79c5c,8893de11e0d23999] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:06:16.607 INFO 26087 --- [ocp-schedules-11,abd9fd7c71e79c5c,8893de11e0d23999] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:06:16.609 INFO 26087 --- [ocp-schedules-25,edb2833a8d5e2e00,6b1be091d8815ca0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:06:16.616 INFO 26087 --- [ocp-schedules-5,6556dea9607b75b6,8fb4d2051f602f41] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:06:16.618 INFO 26087 --- [ocp-schedules-12,67f40af24c652545,2411582658ff74cb] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:06:16.618 INFO 26087 --- [ocp-schedules-12,67f40af24c652545,2411582658ff74cb] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:06:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:06:16.623557+08:00, lastProcessTime=2025-04-08T21:06:06.623849+08:00 2025-04-08 21:06:16.628 INFO 26087 --- [ocp-schedules-29,c2e259b3270bee33,d050764e2601ec70] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:06:16.648 INFO 26087 --- [ocp-schedules-9,4ce5e59f465f7272,0dc2fe61678f21b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:06:16.650 INFO 26087 --- [ocp-schedules-9,4ce5e59f465f7272,0dc2fe61678f21b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:06:16.650 INFO 26087 --- [ocp-schedules-9,4ce5e59f465f7272,0dc2fe61678f21b6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:06:16.742 INFO 26087 --- [ocp-schedules-6,38b4d9b6b9221f83,ba98a6d379f14976] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:06:16.749 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:06:16.623557+08:00 2025-04-08 21:06:16.754 INFO 26087 --- [ocp-schedules-6,38b4d9b6b9221f83,ba98a6d379f14976] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:06:16.754 INFO 26087 --- [ocp-schedules-6,38b4d9b6b9221f83,ba98a6d379f14976] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:06:16.793 INFO 26087 --- [ocp-schedules-21,27b51bc2c1735cac,12b244dea4349269] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:06:16.793 INFO 26087 --- [ocp-schedules-21,27b51bc2c1735cac,12b244dea4349269] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:06:16.793 INFO 26087 --- [ocp-schedules-21,27b51bc2c1735cac,12b244dea4349269] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:06:16.793 INFO 26087 --- [ocp-schedules-21,27b51bc2c1735cac,12b244dea4349269] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:06:16.794 INFO 26087 --- [ocp-schedules-21,27b51bc2c1735cac,12b244dea4349269] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:06:16.794 INFO 26087 --- [ocp-schedules-21,27b51bc2c1735cac,12b244dea4349269] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:06:16.911 INFO 26087 --- [ocp-schedules-17,d77426c532379dff,b3862384a1ff5317] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:06:16.912 INFO 26087 --- [ocp-schedules-17,d77426c532379dff,b3862384a1ff5317] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:06:17.158 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:17.159 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:17.478 INFO 26087 --- [ocp-schedules-20,c944b47b22f7201d,c868e7adf9c6f9c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:06:18.062 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:06:18.062 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:06:18.065 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:06:18.065 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:06:18.065 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:06:18.065 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:06:18.072 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:06:18.072 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:06:18.072 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:06:18.072 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:06:18.096 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:06:18.101 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:06:18.101 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:06:18.101 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:06:18.101 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:06:18.101 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:06:18.106 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:06:18.355 INFO 26087 --- [prometheus-minute-schedule-1,c5f33025a132e206,708128de6a74a6d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:18.436 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:06:18.557 WARN 26087 --- [ocp-async-3,0f42322af5824f22,48aa08fc8c9ef1fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:18.557 INFO 26087 --- [ocp-async-2,12d3ea5bc4dd177c,e82556c039329c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:18.757 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.685519283S 2025-04-08 21:06:18.758 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:06:18.813 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:06:18.813 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:06:18.839 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:06:18.880 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:06:18.880 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:06:18.880 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:06:18.880 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:06:18.880 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:06:18.880 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:06:18.888 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:06:18.986 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] o.s.boot.SpringApplication : Started application in 0.921 seconds (JVM running for 284.359) 2025-04-08 21:06:18.986 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.92124041S 2025-04-08 21:06:19.277 INFO 26087 --- [ocp-schedules-6,5eca29c194bdeca7,e408d7c40df91923] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:06:20.159 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:20.160 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:20.557 WARN 26087 --- [ocp-async-2,dd7832a230483d33,98250d2053caceaf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:20.557 INFO 26087 --- [ocp-async-2,4a8dbf3f8e37df41,9d9357c96d0ffcc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:06:21.355 INFO 26087 --- [prometheus-minute-schedule-1,077b50f98a932b1b,922ba5353e54fc10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:22.557 WARN 26087 --- [ocp-async-5,f7bdb7440047640d,7dfb081aefd730eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:22.557 INFO 26087 --- [ocp-async-2,ad7ffcfd38995939,5eacd187e8857217] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:23.160 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:23.160 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:24.355 INFO 26087 --- [prometheus-minute-schedule-1,2f9914fcf9e66c29,38e3b1075e424f95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:24.557 WARN 26087 --- [ocp-async-3,13cb908c11d22d2b,65ceeb50834a1ca1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:24.557 INFO 26087 --- [ocp-async-9,7e440d0548c32437,5087e4900908fcc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:26.161 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:26.161 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:26.540 INFO 26087 --- [ocp-async-6,4f2ea0e76d915efa,8c7b9fdd18631e67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:06:26.557 WARN 26087 --- [ocp-async-7,413e50ca6aa24259,ad487aee50d4a020] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:26.557 INFO 26087 --- [ocp-async-5,be96c8b5f743bdbc,60101e1f0e0505d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:26.576 INFO 26087 --- [ocp-schedules-26,0964216fd11bacdf,2222b44b1dcbfa61] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:06:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:06:26.623767+08:00, lastProcessTime=2025-04-08T21:06:16.623557+08:00 2025-04-08 21:06:26.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:06:26.623767+08:00 2025-04-08 21:06:26.919 INFO 26087 --- [ocp-schedules-6,6de705727135be9d,f1d76544594cb523] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:06:26.920 INFO 26087 --- [ocp-schedules-6,6de705727135be9d,f1d76544594cb523] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:06:27.355 INFO 26087 --- [prometheus-minute-schedule-1,7722c5a894d89307,78557173beae1d4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:27.537 INFO 26087 --- [ocp-schedules-3,619348bf0858bf10,7bf3f62cce883d6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:06:28.557 WARN 26087 --- [ocp-async-5,ba43298be68dc19f,9ecaa78543e0b28d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:28.557 INFO 26087 --- [ocp-async-3,31742dd52a78d394,46dc0ea2927c7786] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:29.161 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:29.162 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:30.355 INFO 26087 --- [prometheus-minute-schedule-1,ffa8d9823c861170,01172cdcc2559662] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:30.558 WARN 26087 --- [ocp-async-9,1958d7c33ebfc335,d24b3042832e77a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:30.558 INFO 26087 --- [ocp-async-8,e9fbb386ef24131e,b51174981d1d6f21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:32.162 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:32.163 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:32.558 WARN 26087 --- [ocp-async-10,f6022a5f505b9f48,787906889d9f4834] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:32.558 INFO 26087 --- [ocp-async-7,2596ee0f3f7fa2f0,d093c46417aef8d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:33.355 INFO 26087 --- [prometheus-minute-schedule-1,5430c5266efd8463,76660cad38b7d53e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:34.558 WARN 26087 --- [ocp-async-5,e639f46d30369267,a92ed5a62e7f50fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:34.558 INFO 26087 --- [ocp-async-6,ff5e077e7580d5bf,7666219691031302] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:35.163 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:35.164 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:36.355 INFO 26087 --- [prometheus-minute-schedule-1,f8155cdcaea6ad3c,b5448bbca6a5e889] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:36.540 INFO 26087 --- [ocp-async-3,e228e1f1852c4ca7,254dbb4c72ec7355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:06:36.555 INFO 26087 --- [ocp-schedules-30,5f5c2d9bbc1ed15e,e31f211dbd7e69f6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:06:36.558 WARN 26087 --- [ocp-async-9,a8f587620eee547b,24c44664ae3a7f3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:36.558 INFO 26087 --- [ocp-async-2,5f074a42e6a2b1e2,5da8bb3b2963fc90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:36.577 INFO 26087 --- [ocp-schedules-13,76f7a24e02dffc84,d2cb59a28f1ea4bd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:06:36.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:06:36.624511+08:00, lastProcessTime=2025-04-08T21:06:26.623767+08:00 2025-04-08 21:06:36.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:06:36.624511+08:00 2025-04-08 21:06:36.755 INFO 26087 --- [ocp-schedules-20,48f0afd47c4b010d,ccafd299696b6fe6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:06:36.766 INFO 26087 --- [ocp-schedules-20,48f0afd47c4b010d,ccafd299696b6fe6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:06:36.766 INFO 26087 --- [ocp-schedules-20,48f0afd47c4b010d,ccafd299696b6fe6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:06:36.924 INFO 26087 --- [ocp-schedules-22,8fcbcf785d7c066d,47eda7dd41a2348f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:06:36.925 INFO 26087 --- [ocp-schedules-22,8fcbcf785d7c066d,47eda7dd41a2348f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:06:37.583 INFO 26087 --- [ocp-schedules-27,d30859d166996d6f,d8be7b275f3c88fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:06:38.164 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:38.164 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:38.558 WARN 26087 --- [ocp-async-10,d4e6f32d6578a93a,039d837782d69b49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:38.558 INFO 26087 --- [ocp-async-4,cad21ca3437e7f3b,51ad5360f6d0d027] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:39.355 INFO 26087 --- [prometheus-minute-schedule-1,84185596a375adb5,57644b59c232d353] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:40.001 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:06:40.001 INFO 26087 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:06:40.558 WARN 26087 --- [ocp-async-1,dd4a4213f7e70518,da0df0b2cda841d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:40.558 INFO 26087 --- [ocp-async-6,82fe549a4bb31577,44f26e5e43ed34bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:41.164 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:41.165 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:42.355 INFO 26087 --- [prometheus-minute-schedule-1,a61b8368903ba0f2,89acdc68e1c7dcb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:42.559 WARN 26087 --- [ocp-async-9,31ef17c5dc964ff4,b956e452a80da043] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:42.559 INFO 26087 --- [ocp-async-8,5fe85f90af27d9b9,0b3e82e94378c11d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:44.165 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:44.166 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:44.559 WARN 26087 --- [ocp-async-7,856b28b830d30013,327d47b8ec1b523c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:44.559 INFO 26087 --- [ocp-async-10,558a9ec0bfd324ff,b7051f9786cc4768] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:45.355 INFO 26087 --- [prometheus-minute-schedule-1,f084e632c516e515,d2854a827bd86b04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:46.521 INFO 26087 --- [ocp-schedules-24,21be3883f7bef330,dde7f87d4455591f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:06:46.521 INFO 26087 --- [ocp-schedules-16,acd0b21abe1a8fe3,e76959d3d9d19cdf] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:06:46.521 INFO 26087 --- [ocp-schedules-16,acd0b21abe1a8fe3,e76959d3d9d19cdf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:06:46.523 INFO 26087 --- [ocp-schedules-8,b8afed5458d08aaf,a49c1ea34f1baef9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:06:46.523 INFO 26087 --- [ocp-schedules-8,b8afed5458d08aaf,a49c1ea34f1baef9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:06:46.523 INFO 26087 --- [ocp-schedules-27,03db07fb80a64294,fea3b8e53385ab9d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:06:46.524 INFO 26087 --- [ocp-schedules-27,03db07fb80a64294,fea3b8e53385ab9d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:06:46.524 INFO 26087 --- [ocp-schedules-28,4f619414615163ba,b91887c56b29d2ae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:06:46.524 INFO 26087 --- [ocp-schedules-25,128b201ab32dff0b,4e5bfe68e52f0d9b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:06:46.524 INFO 26087 --- [ocp-schedules-22,e669f2dd5c4bfb4c,fcebc55602f63e18] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:06:46.525 INFO 26087 --- [ocp-schedules-11,1f17c73c27b7061e,400cd40a5fb7f1be] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:06:46.541 INFO 26087 --- [ocp-async-4,01eec447fa63ec33,60ff936397b21f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:06:46.559 WARN 26087 --- [ocp-async-6,9ba9c97c05a544b4,6dcfd9dbcd7f920e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:46.559 INFO 26087 --- [ocp-async-1,f91b3fef5ffd1cc6,a0f7c8e18ec3f56d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:46.578 INFO 26087 --- [ocp-schedules-12,57b47156aa1a7855,f9572e72e71fb133] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:06:46.578 INFO 26087 --- [ocp-schedules-3,16605afda01eb879,3234c51574447293] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:06:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:06:46.623327+08:00, lastProcessTime=2025-04-08T21:06:36.624511+08:00 2025-04-08 21:06:46.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:06:46.623327+08:00 2025-04-08 21:06:46.956 INFO 26087 --- [ocp-schedules-4,8b4e29cb43e10627,1bf1543186727bbf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:06:46.957 INFO 26087 --- [ocp-schedules-4,8b4e29cb43e10627,1bf1543186727bbf] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:06:47.166 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:47.167 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:47.608 INFO 26087 --- [ocp-schedules-25,3f9514a802e60c62,ab9144e5dc9a9aa2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:06:48.355 INFO 26087 --- [prometheus-minute-schedule-1,8e40c88fce6d34f6,8ecfc7f4e70c1512] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:48.559 WARN 26087 --- [ocp-async-3,1e9232fa190fbc60,11d15d404c309906] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:48.559 INFO 26087 --- [ocp-async-9,c81b9fc8454af40c,52e254153ed47f7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:50.167 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:50.167 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:50.559 WARN 26087 --- [ocp-async-7,7805eca0a3455733,842b733117b84d00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:50.559 INFO 26087 --- [ocp-async-10,ed6a8dcb2c24f884,c659583f2ce14aad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:51.355 INFO 26087 --- [prometheus-minute-schedule-1,7d03257d5e524ca4,e6e19db7b86fbe80] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:52.559 WARN 26087 --- [ocp-async-6,117d1728df7369a2,616de3d144ce6cde] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:52.559 INFO 26087 --- [ocp-async-4,cb20f0713a9f9d65,e4c8edfad9d3048d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:52.735 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials, client 10.11.12.102, traceId 794ef96014b2efca] 2025-04-08 21:06:52.750 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials?null, client=10.11.12.102, traceId=794ef96014b2efca, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.createCredential(CreateCredentialParam), args=CreateCredentialParam(targetType=HOST, validate=true, name=adminpingzheng, description=, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=admin, passphrase=xxx hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null),] 2025-04-08 21:06:52.755 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=HOST, validate=true, name=adminpingzheng, description=, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=admin, passphrase=Xxf7713123#$^%, hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null) 2025-04-08 21:06:52.756 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.o.s.c.CredentialServiceImpl : Validating credential:ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=admin, passphrase=Xxf7713123#$^%, hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null, batchSshValidateProperty=null) 2025-04-08 21:06:52.781 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=HOST, validate=true, name=adminpingzheng, description=, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=null, tenantId=null, clusterName=null, tenantName=null, username=null), sshCredentialProperty=SshCredentialProperty(type=PASSWORD, allowSudo=true, username=admin, passphrase=Xxf7713123#$^%, hostIdList=[], localPrivateKeyPath=null), agentCredentialProperty=null) 2025-04-08 21:06:52.783 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials client=10.11.12.102, traceId=794ef96014b2efca, duration=51 ms] 2025-04-08 21:06:52.785 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,794ef96014b2efca,e13ccb39975b511d] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=CREATE_CREDENTIAL, eventType=CREDENTIAL_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.CredentialController.createCredential, resourceType=CREDENTIAL, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=adminpingzheng, parentResource=null)], operatorId=100, operatorName=admin, status=SUCCESS, failedReason=null, taskId=null, eventInfo={"param":{"description":"","name":"adminpingzheng","obJdbcCredentialProperty":{},"sshCredentialProperty":{"allowSudo":true,"hostIdList":[],"passphrase":"******","type":"PASSWORD","username":"admin"},"targetType":"HOST","validate":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/profiles/me/credentials, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T21:06:52.750+08:00, endTime=2025-04-08T21:06:52.784+08:00, role=null, aliUid=null) 2025-04-08 21:06:52.914 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,cd38a71932a9a1bc,1ab7658e58f56981] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=cd38a71932a9a1bc, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 21:06:52.918 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,cd38a71932a9a1bc,1ab7658e58f56981] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=cd38a71932a9a1bc, duration=8 ms] 2025-04-08 21:06:53.168 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:53.168 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:54.355 INFO 26087 --- [prometheus-minute-schedule-1,ef8b23080b6c81fa,631f02d63716ce75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:54.559 WARN 26087 --- [ocp-async-4,5e9d4b2c5ef4cdf7,6835b969da284f99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:54.560 INFO 26087 --- [ocp-async-8,c8147b1afb468eae,a5b7eea4ba3a5ab1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:56.168 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:56.169 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:06:56.541 INFO 26087 --- [ocp-async-9,03c84a68d541d3c7,2cbb7bb8a1eca44c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:06:56.541 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,0ed83ed46fea8359,1bea034d9006ed89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=0ed83ed46fea8359, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:06:56.560 WARN 26087 --- [ocp-async-10,3a2f09d17cc31eda,3e104b411597e34c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:56.560 INFO 26087 --- [ocp-async-2,a06123c7a99ea2d3,cae02c03de97dcb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:56.579 INFO 26087 --- [ocp-schedules-9,b3d8c04b08de7b29,6ea9e63f9ed231ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:06:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:06:56.622222+08:00, lastProcessTime=2025-04-08T21:06:46.623327+08:00 2025-04-08 21:06:56.632 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,0ed83ed46fea8359,1bea034d9006ed89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=0ed83ed46fea8359, duration=96 ms] 2025-04-08 21:06:56.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:06:56.622222+08:00 2025-04-08 21:06:56.767 INFO 26087 --- [ocp-schedules-15,dae8f94ca5c62d9a,3cb4e35079e5f408] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:06:56.771 INFO 26087 --- [ocp-schedules-15,dae8f94ca5c62d9a,3cb4e35079e5f408] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:06:56.771 INFO 26087 --- [ocp-schedules-15,dae8f94ca5c62d9a,3cb4e35079e5f408] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:06:56.990 INFO 26087 --- [ocp-schedules-25,d0dbfa1487902a08,39bb9ccdc409e4e9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:06:56.992 INFO 26087 --- [ocp-schedules-25,d0dbfa1487902a08,39bb9ccdc409e4e9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:06:57.355 INFO 26087 --- [prometheus-minute-schedule-1,dfda2de6cc5f16df,bcc58a7c46747889] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117620, interval=60 2025-04-08 21:06:57.630 INFO 26087 --- [ocp-schedules-2,c8c640da012b2230,0feacc61b3e59b2a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:06:58.560 WARN 26087 --- [ocp-async-6,276018297d8329b1,eb7aabac3e2c0c8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:06:58.560 INFO 26087 --- [ocp-async-5,61fd0041a5b9cc79,c13c1bfd5dbe435d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:06:59.169 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:06:59.170 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:00.355 INFO 26087 --- [prometheus-minute-schedule-1,d37c7b66fe64eb58,0c3befa6f1265f7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:00.560 WARN 26087 --- [ocp-async-8,739f30351b1a84f0,d447b18d8661b955] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:00.560 INFO 26087 --- [ocp-async-4,76bffddec17c3f1e,17aaf2ab8873f963] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:02.170 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:02.171 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:02.560 WARN 26087 --- [ocp-async-7,787cf63e34b7bc2c,ecd6af8a63d427ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:02.560 INFO 26087 --- [ocp-async-9,513002e80b79ca39,56eb9bcb16658760] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:03.355 INFO 26087 --- [prometheus-minute-schedule-1,e9cd9331022a7533,6d593f0f3b3c49e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:04.560 WARN 26087 --- [ocp-async-2,f8eeea7c3abf304f,b8fdc349a5242fdb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:04.560 INFO 26087 --- [ocp-async-1,0b1781acb07da7e0,0e1e869aa21acaa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:05.171 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:05.171 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:05.307 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,b8cadc1683299195,bc8b8e43176e0e02] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId b8cadc1683299195] 2025-04-08 21:07:05.314 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,b8cadc1683299195,bc8b8e43176e0e02] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=b8cadc1683299195, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1000001, localPrivateKeyPath=null)),] 2025-04-08 21:07:05.321 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,b8cadc1683299195,bc8b8e43176e0e02] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=admin, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)] 2025-04-08 21:07:05.323 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 21:07:05.329 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 21:07:05.336 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 21:07:05.349 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.61:22 success 2025-04-08 21:07:05.365 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: admin@192.168.1.61 2025-04-08 21:07:05.365 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.61, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=admin), privateKeyAuthConfig=null)), proxy=null), estimated size:2 2025-04-08 21:07:05.413 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:05.416 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:05.456 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:05.459 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:05.459 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:05.459 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:05.462 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:05.462 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.61 2025-04-08 21:07:05.476 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:05.476 INFO 26087 --- [pool-22-thread-1,b8cadc1683299195,35ca9ef6b7fac159] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:05.477 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,b8cadc1683299195,bc8b8e43176e0e02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=b8cadc1683299195, duration=173 ms] 2025-04-08 21:07:05.555 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOver, client 10.11.12.102, traceId 34015dfa7d12dcff] 2025-04-08 21:07:05.565 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOver?null, client=10.11.12.102, traceId=34015dfa7d12dcff, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster(TakeOverClusterParam), args=TakeOverClusterParam(switchConfigUrl=true, hostInfo=TakeOverClusterParam.HostInfo(kind=DEDICATED_PHYSICAL_MACHINE, publishPorts=null, hostTypeId=1, sshPort=22, credentialId=1000001)),] 2025-04-08 21:07:05.574 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-192.168.1.61#2881, method=takeOverCluster 2025-04-08 21:07:05.585 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.586 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.586 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.587 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.587 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.588 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.588 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.588 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.589 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.589 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.593 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.593 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 21:07:05.599 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.599 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.600 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.600 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 21:07:05.605 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.605 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.606 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.606 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 21:07:05.611 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.611 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.612 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.612 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.616 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.616 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 21:07:05.617 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.617 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.619 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.619 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 21:07:05.623 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.624 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.627 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.628 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 21:07:05.628 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.629 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.632 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.634 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 21:07:05.635 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.636 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.636 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.637 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.640 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:07:05.640 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.643 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:07:05.647 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.648 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.651 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.652 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:07:05.666 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.667 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.667 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.668 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.668 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.669 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.669 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.669 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:07:05.686 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.686 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.690 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.690 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:07:05.691 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.692 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.692 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.693 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:07:05.709 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.710 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.710 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.711 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.711 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.712 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.712 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.713 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.713 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.713 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:07:05.728 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.729 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.729 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.730 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:07:05.730 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.731 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.731 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.732 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:07:05.747 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.748 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.749 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.749 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.749 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.750 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.750 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.751 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.751 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.752 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:07:05.768 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.769 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.770 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.770 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:07:05.774 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.775 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.775 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.777 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.777 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.778 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 21:07:05.787 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.787 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.789 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.789 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.790 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.790 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:07:05.791 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.791 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:07:05.791 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.792 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:07:05.796 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.796 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.797 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.797 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.798 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.798 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:05.800 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:05.800 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:07:05.807 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 21:07:05.830 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.host.service.RegionServiceImpl : Created region: 1000002 deault_region 2025-04-08 21:07:05.857 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.host.service.IdcServiceImpl : Created idc: 1000001 default_idc 2025-04-08 21:07:05.914 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000001, resourceId=1000001 2025-04-08 21:07:05.936 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000001, resourceId=1000001 2025-04-08 21:07:05.990 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000001, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:07:06.043 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000001, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:07:06.043 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000001, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:07:06.044 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.CredentialServiceImpl : Credential clusterName=ocpdb|tenantName=sys|username=root is already exists 2025-04-08 21:07:06.044 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.CredentialServiceImpl : Delete credential:clusterName=ocpdb|tenantName=sys|username=root 2025-04-08 21:07:06.048 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000001, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:07:06.049 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.049 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:07:06.050 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.050 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:07:06.050 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.051 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:07:06.055 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.055 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:06.056 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.056 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:06.057 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.057 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:06.057 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.058 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:06.058 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:06.058 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:07:06.063 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.o.i.c.ClusterOperationServiceImpl : meta cluster detected. 2025-04-08 21:07:06.091 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE APP GROUP, property:UpdateAppGroupProperty(serialNumbers=[null], appGroup=null) 2025-04-08 21:07:06.092 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000004, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:07:06.093 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Take over primary OB cluster, nodeCount=39, taskCount=39 2025-04-08 21:07:06.093 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000001, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, createTime=null, startTime=2025-04-08T21:07:06.093+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update zone arch, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.UpdateZoneArchTask, seriesId=16, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=28, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Register config url, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RegisterConfigUrlTask, seriesId=7, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Sync all tenant information, state=PENDING, operation=EXECUTE, 2025-04-08 21:07:06.146 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000038} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.166 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000037} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1000001, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.167 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000016} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.168 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000019} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.168 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000040} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.169 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000036} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.172 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000007} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1000001, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000001, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.173 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000042} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1000001, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000001, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.173 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000030} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.174 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000039} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.175 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000031} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.175 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000041} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000001, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:07:06.178 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=1000006, taskName=Take over primary OB cluster, argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}} 2025-04-08 21:07:06.200 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOver client=10.11.12.102, traceId=34015dfa7d12dcff, duration=647 ms] 2025-04-08 21:07:06.201 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,34015dfa7d12dcff,a91379c8f79c1e6c] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=1000006, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1000001,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T21:07:05.565+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 21:07:06.355 INFO 26087 --- [prometheus-minute-schedule-1,89a9d69abc4a8e1d,652ca946793c637d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:06.355 INFO 26087 --- [prometheus-cache-monitor1,61270c5a6bbe6bd8,18c639a564d04aa4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":364,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":366,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":364} 2025-04-08 21:07:06.355 INFO 26087 --- [prometheus-cache-monitor1,61270c5a6bbe6bd8,18c639a564d04aa4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:07:06.533 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 21:07:06.533 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:07:06.533 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 21:07:06.541 INFO 26087 --- [ocp-async-5,dd14da2799eb5a93,337fa7e763212399] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:07:06.541 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db success. 2025-04-08 21:07:06.543 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000004, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:07:06.560 WARN 26087 --- [ocp-async-8,2538f23d22aabf2f,6c24f0e2900308aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:06.560 INFO 26087 --- [ocp-async-4,8ff913321185d4b6,ead2781e12d86ef2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:06.580 INFO 26087 --- [ocp-schedules-28,ebecb6510f27ff9d,9b0fae8b705bca26] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:07:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:07:06.623289+08:00, lastProcessTime=2025-04-08T21:06:56.622222+08:00 2025-04-08 21:07:06.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 21:07:06.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:07:06.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 21:07:06.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:07:06.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:07:06.717 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:07:06.623289+08:00 2025-04-08 21:07:06.797 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,e61ff1e9a7c05110] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000034, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:06.808 INFO 26087 --- [manual-subtask-executor14,34015dfa7d12dcff,7d5615c0518be7d9] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:06.820 INFO 26087 --- [manual-subtask-executor14,34015dfa7d12dcff,7d5615c0518be7d9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000034, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, service_name=ocpdb:1744114127, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, latest_execution_start_time=2025-04-08T21:07:06.792+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=1000034}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:06.827 INFO 26087 --- [manual-subtask-executor14,34015dfa7d12dcff,7d5615c0518be7d9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000034, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:06.837 INFO 26087 --- [manual-subtask-executor14,34015dfa7d12dcff,7d5615c0518be7d9] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000040] 2025-04-08 21:07:07.000 INFO 26087 --- [ocp-schedules-11,95f4af43ad57b2e1,3d98e78c697c1928] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:07:07.001 INFO 26087 --- [ocp-schedules-11,95f4af43ad57b2e1,3d98e78c697c1928] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:07:07.351 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000004 2025-04-08 21:07:07.655 INFO 26087 --- [ocp-schedules-9,1ab94a8da6dc576b,0147d0b374a1415e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:07:07.805 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,9c47424a175b861a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000040, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:07.823 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:07.827 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000040, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:07.804+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000040}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:07.846 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:07.852 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:07.868 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.61 2025-04-08 21:07:07.890 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:07.899 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:07.940 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.61 2025-04-08 21:07:07.946 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:07.948 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:07.955 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 21:07:07.980 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.61, port=22, loopCount=1, minSuccessCount=1 2025-04-08 21:07:07.987 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:07.999 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:08.039 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.61 2025-04-08 21:07:08.048 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=whoami, out=admin, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:08.056 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=whoami, out=admin, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:08.066 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[79] 2025-04-08 21:07:08.079 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=79, maxAcceptableDelayMillis=3000 2025-04-08 21:07:08.092 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.61], timeoutMillis=20000 2025-04-08 21:07:08.124 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744117628 0 0 , error= 2025-04-08 21:07:08.126 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 21:07:08.147 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000040, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:08.149 INFO 26087 --- [manual-subtask-executor15,34015dfa7d12dcff,0fbd6107513ab16e] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000043] 2025-04-08 21:07:08.172 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:08.172 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:08.561 WARN 26087 --- [ocp-async-7,a29648a57a5bed1a,9e65ca1fc5f48eb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:08.561 INFO 26087 --- [ocp-async-10,ff35ed37d63bfb93,518afbfa71de57d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:08.821 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,3bb5349c2f4af370] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000043, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:08.893 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,00160ee4dd6e7cd9] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:08.901 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,00160ee4dd6e7cd9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000043, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:08.820+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000043}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:08.912 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,00160ee4dd6e7cd9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000043, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:08.914 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,00160ee4dd6e7cd9] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000041] 2025-04-08 21:07:09.355 INFO 26087 --- [prometheus-minute-schedule-1,8a0fa19db7460a04,2ecaa64668e63d1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:09.874 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,e16d76e844907217] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000041, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:09.900 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:09.904 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000041, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:09.872+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000041}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:09.908 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 21:07:09.933 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:09.948 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:09.988 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:09.995 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:10.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:07:10.001 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:07:10.003 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.61 2025-04-08 21:07:10.023 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:10.027 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:10.067 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 21:07:10.096 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000041, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:10.098 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,deb45693516b57b7] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000033] 2025-04-08 21:07:10.561 WARN 26087 --- [ocp-async-6,c4b760cbc2c76fd1,6640f61eda48ea8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:10.561 INFO 26087 --- [ocp-async-2,967e2601a7a5b4cf,94fda0d2092ef65f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:10.605 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:07:10.896 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,fe1534e22ff2236f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000033, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:10.906 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f828298da9b5d61f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:10.924 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f828298da9b5d61f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000033, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:10.895+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000033}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:10.944 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f828298da9b5d61f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000033, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:10.946 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f828298da9b5d61f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000031] 2025-04-08 21:07:11.172 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:11.173 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:11.905 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,18720348f84a2647] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000031, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:11.932 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:11.937 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000031, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:11.904+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000031}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:11.945 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 21:07:11.956 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:11.963 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:11.982 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:11.993 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.033 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:12.058 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.070 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.081 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:12.095 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.135 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61 2025-04-08 21:07:12.327 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117560, nodeCount=366, affectRows=366 2025-04-08 21:07:12.330 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117560, nodeCount=365, validCount=365 2025-04-08 21:07:12.355 INFO 26087 --- [prometheus-minute-schedule-1,faa75141e349d2d5,6458ca67b96ad52c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:12.561 WARN 26087 --- [ocp-async-8,8ff5c05cfd452309,917d7bc07359b1ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:12.561 INFO 26087 --- [ocp-async-3,c8127507e329c98e,fcf52498d59e0a62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:12.487 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.588 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.610 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 21:07:12.620 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 21:07:12.707 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:12.753 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:12.813 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61 2025-04-08 21:07:13.213 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:13.223 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:13.225 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 21:07:13.231 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 21:07:13.245 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 21:07:13.281 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000031, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:13.283 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,41db5b7343cb26ff] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000026] 2025-04-08 21:07:13.931 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,a727029c4e517fa8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000026, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:13.956 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,ea04f72b2583a4c6] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:13.959 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,ea04f72b2583a4c6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000026, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:13.929+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000026}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:13.990 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,ea04f72b2583a4c6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000026, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:13.992 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,ea04f72b2583a4c6] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000030] 2025-04-08 21:07:14.173 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:14.174 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:14.561 WARN 26087 --- [ocp-async-3,2a8cc788b3640073,aa937d556c5fac7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:14.561 INFO 26087 --- [ocp-async-4,c62ac6dbf6a2a58a,6e77e8bf6881fdfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:14.955 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,99603148948d33d2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000030, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:14.965 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:14.984 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000030, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:14.953+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000030}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:14.991 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 21:07:15.005 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.014 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.054 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.074 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.077 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.091 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.093 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.131 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.150 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.154 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.174 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.184 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.214 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.229 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.232 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.259 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.271 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.299 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.305 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.308 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.325 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.331 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.355 INFO 26087 --- [prometheus-minute-schedule-1,b137819b3eec6483,b8308ba9c9fb714d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:15.365 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.370 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.373 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.385 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.388 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.426 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.435 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.438 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.450 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.452 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.491 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.497 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.500 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.513 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.523 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.553 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.560 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.563 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.575 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.577 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.615 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.642 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.648 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:15.663 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.671 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.703 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:15.708 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:15.714 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:07:16.084 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.086 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.088 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:07:16.090 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:07:16.124 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:16.130 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.133 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:16.147 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.152 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.187 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:16.191 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.195 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:16.213 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.225 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.254 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:16.268 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.271 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:07:16.294 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.300 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.302 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:07:16.335 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:16.341 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.349 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:07:16.517 INFO 26087 --- [ocp-schedules-29,9a1328a10ff52d0b,9aafbf5ef2277867] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:07:16.517 INFO 26087 --- [ocp-schedules-24,be7728e7eb4f4942,564b3f435b2df11a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:07:16.521 INFO 26087 --- [ocp-schedules-20,9e97567eba0adab6,b9a48147c9115a24] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:07:16.522 INFO 26087 --- [ocp-schedules-22,0c373197b55adb3a,63e08bd248c1c125] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:07:16.522 INFO 26087 --- [ocp-schedules-22,0c373197b55adb3a,63e08bd248c1c125] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-11,ddfb70a5dba8a529,a1a772549769501e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-11,ddfb70a5dba8a529,a1a772549769501e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-24,be7728e7eb4f4942,564b3f435b2df11a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-16,a6bb1319b062102e,de4ee127dcbf50ee] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-16,a6bb1319b062102e,de4ee127dcbf50ee] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-16,c224ec4d18c503da,3c8bc6041f1a0752] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-27,3f8f8f3968af149d,984b6e91fcf9f5ca] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:07:16.524 INFO 26087 --- [ocp-schedules-12,25d4de4ceb9d0a8f,8f75011e2e9fe216] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:07:16.525 INFO 26087 --- [ocp-schedules-16,c224ec4d18c503da,3c8bc6041f1a0752] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:16.525 INFO 26087 --- [ocp-schedules-27,3f8f8f3968af149d,984b6e91fcf9f5ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:16.525 INFO 26087 --- [ocp-schedules-12,25d4de4ceb9d0a8f,8f75011e2e9fe216] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:07:16.530 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000001, status=TAKINGOVER 2025-04-08 21:07:16.530 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:07:16.531 INFO 26087 --- [ocp-schedules-29,9a1328a10ff52d0b,9aafbf5ef2277867] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:07:16.541 INFO 26087 --- [ocp-async-6,af4ffffb693ce5b8,e12befbd4a8404c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:07:16.543 INFO 26087 --- [ocp-schedules-1,3f8e04d7950d980f,f89e02bc05c8da7d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:07:16.544 INFO 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:07:16.544 INFO 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2025-04-08 21:07:16.548 INFO 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:07:16.550 WARN 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:07:16.550 INFO 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:07:16.552 WARN 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:07:16.553 INFO 26087 --- [ocp-schedules-6,027888eb892a2f7c,bbdfeb9f2304e597] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:07:16.555 INFO 26087 --- [ocp-schedules-6,027888eb892a2f7c,bbdfeb9f2304e597] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:07:16.556 WARN 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:07:16.557 ERROR 26087 --- [ocp-schedules-3,337b614939ff0cff,0ae0b593dff087ac] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1000001 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1788) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1669) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted 2025-04-08 21:07:16.562 WARN 26087 --- [ocp-async-8,773d55af36a339b0,ee56edce8a76a083] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:16.562 INFO 26087 --- [ocp-async-2,7f1059b1bc2374af,76b73ce529c1e71a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:16.576 INFO 26087 --- [ocp-schedules-4,26ca502363eb270e,a3c8d1b3df4d9610] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:07:16.580 INFO 26087 --- [ocp-schedules-21,3a64fd0b3a438db3,9aea643413168458] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:07:16.580 INFO 26087 --- [ocp-schedules-2,d707628ed4b07ae2,cfab02620c2829a2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:07:16.580+08:00, clusters=[1000001] 2025-04-08 21:07:16.582 INFO 26087 --- [ocp-schedules-20,3176f9410fd19fe9,3700b8a184c537d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:07:16.582 INFO 26087 --- [ocp-schedules-2,d707628ed4b07ae2,cfab02620c2829a2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:07:16.583 INFO 26087 --- [ocp-schedules-22,5fa5a9c60b4faed7,c7d3e8f73e526f39] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:07:16.583+08:00, clusters=[1000001] 2025-04-08 21:07:16.583 INFO 26087 --- [ocp-schedules-26,7c1d0e3a25b70cbf,7053832c58162bdc] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:07:16.583 INFO 26087 --- [ocp-schedules-22,5fa5a9c60b4faed7,c7d3e8f73e526f39] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:07:16.584 INFO 26087 --- [ocp-schedules-24,82018ca44c397c74,ea8196714926b215] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":730,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":372,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":366} 2025-04-08 21:07:16.584 INFO 26087 --- [ocp-schedules-24,82018ca44c397c74,ea8196714926b215] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:07:16.595 INFO 26087 --- [ocp-schedules-28,22a6dbb5affd600b,ec6fd90a6df3175a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:07:16.595 INFO 26087 --- [ocp-schedules-28,22a6dbb5affd600b,ec6fd90a6df3175a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:07:16.607 INFO 26087 --- [ocp-schedules-5,88b1fbba6db3b920,a58bbd2bc98b9899] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:07:16.607 INFO 26087 --- [ocp-schedules-5,88b1fbba6db3b920,a58bbd2bc98b9899] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:07:16.610 INFO 26087 --- [ocp-schedules-29,20b54796edbb09d1,a3e633430f19ed2e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:07:16.610 INFO 26087 --- [ocp-schedules-29,20b54796edbb09d1,a3e633430f19ed2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:16.612 INFO 26087 --- [ocp-schedules-5,88b1fbba6db3b920,a58bbd2bc98b9899] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:07:16.612 INFO 26087 --- [ocp-schedules-5,88b1fbba6db3b920,a58bbd2bc98b9899] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:07:16.619 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:16.619 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:07:16.620 INFO 26087 --- [ocp-schedules-17,433236548ab40015,d457473efc85f2b1] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:07:16.620 INFO 26087 --- [ocp-schedules-17,433236548ab40015,d457473efc85f2b1] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:07:16.621 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:16.621 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:07:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:07:16.623537+08:00, lastProcessTime=2025-04-08T21:07:06.623289+08:00 2025-04-08 21:07:16.625 INFO 26087 --- [ocp-schedules-1,1e0edceaae8eb9f4,14ec0a2320f8cc23] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:07:16.629 INFO 26087 --- [ocp-schedules-30,1ce30bca7a69f4e0,a54d8e89331ed3d4] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:07:16.629 INFO 26087 --- [ocp-schedules-30,1ce30bca7a69f4e0,a54d8e89331ed3d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:16.648 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:07:16.649 ERROR 26087 --- [ocp-schedules-30,1ce30bca7a69f4e0,a54d8e89331ed3d4] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:07:16.649 INFO 26087 --- [ocp-schedules-30,1ce30bca7a69f4e0,a54d8e89331ed3d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:07:16.650 INFO 26087 --- [ocp-schedules-10,c3d993d489d222da,069a13b1a8c6f9d4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:07:16.651 INFO 26087 --- [ocp-schedules-10,c3d993d489d222da,069a13b1a8c6f9d4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:07:16.651 INFO 26087 --- [ocp-schedules-10,c3d993d489d222da,069a13b1a8c6f9d4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:07:16.702 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.707 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:07:16.623537+08:00 2025-04-08 21:07:16.715 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:07:16.725 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:07:16.728 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:07:16.743 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:16.754 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:16.768 INFO 26087 --- [ocp-schedules-15,9b855fe0287029cb,cb1beba6ca4499a0] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:07:16.768 INFO 26087 --- [ocp-schedules-15,9b855fe0287029cb,cb1beba6ca4499a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:16.772 INFO 26087 --- [ocp-schedules-13,003d5a696b443730,faffe373b0a9550b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:07:16.780 ERROR 26087 --- [ocp-schedules-15,9b855fe0287029cb,cb1beba6ca4499a0] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:07:16.780 INFO 26087 --- [ocp-schedules-15,9b855fe0287029cb,cb1beba6ca4499a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:07:16.762 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 21:07:16.792 INFO 26087 --- [ocp-schedules-13,003d5a696b443730,faffe373b0a9550b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:07:16.792 INFO 26087 --- [ocp-schedules-13,003d5a696b443730,faffe373b0a9550b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:07:16.794 INFO 26087 --- [ocp-schedules-6,e827f66a604f080b,f7b4644ef50f0607] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:07:16.794 INFO 26087 --- [ocp-schedules-6,e827f66a604f080b,f7b4644ef50f0607] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:07:16.798 INFO 26087 --- [ocp-schedules-6,e827f66a604f080b,f7b4644ef50f0607] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:07:16.798 INFO 26087 --- [ocp-schedules-6,e827f66a604f080b,f7b4644ef50f0607] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:07:16.800 INFO 26087 --- [ocp-schedules-6,e827f66a604f080b,f7b4644ef50f0607] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:07:16.800 INFO 26087 --- [ocp-schedules-6,e827f66a604f080b,f7b4644ef50f0607] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:07:16.908 INFO 26087 --- [ocp-schedules-3,e0230c96a374ec10,8ba2c0f8b9ad598e] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61, current targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61 2025-04-08 21:07:17.001 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:07:17.001 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.001 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:07:17.001 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:07:17.001 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:07:17.001 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:07:17.002 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:07:17.003 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:07:17.004 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:07:17.005 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:07:17.006 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:07:17.007 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:07:17.008 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:07:17.009 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:07:17.010 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:07:17.011 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:07:17.012 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:07:17.013 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:07:17.014 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:07:17.015 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:07:17.016 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:07:17.017 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:07:17.018 INFO 26087 --- [ocp-schedules-19,3ff3b170928c16a0,ecf817c685f5d231] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:07:17.028 INFO 26087 --- [ocp-schedules-4,16cba2e047d47b91,7d0d0bfa13d0d875] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:07:17.029 INFO 26087 --- [ocp-schedules-4,16cba2e047d47b91,7d0d0bfa13d0d875] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:07:17.131 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:17.148 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:17.153 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:07:17.174 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:17.175 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:17.164 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:07:17.194 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:17.204 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:17.206 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61 2025-04-08 21:07:17.227 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:17.230 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:17.234 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 21:07:17.272 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000030, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:17.274 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,f1f2df50edff25df] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000029] 2025-04-08 21:07:17.683 INFO 26087 --- [ocp-schedules-30,e64332db84e7de73,56c122531d10fb08] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:07:17.966 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,36829362c1e5df38] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000029, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:17.996 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,c9f55949155be338] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:18.002 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,c9f55949155be338] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000029, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:17.964+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000029}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:18.008 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,c9f55949155be338] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000029, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:18.009 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,c9f55949155be338] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000016] 2025-04-08 21:07:18.355 INFO 26087 --- [prometheus-minute-schedule-1,a64f285056a90412,a0c10f30d143bb6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:18.562 WARN 26087 --- [ocp-async-3,b1cc3adf64354133,cd12d2526953718a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:18.562 INFO 26087 --- [ocp-async-2,3b667921e055eba6,73e0410674d668a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:18.992 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,56fa714261c69ee3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000016, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:19.011 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:19.015 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000016, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:18.990+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000016}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:19.029 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=50b69f61-2806-4a02-a4c9-581269bae9ac, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000001, authType=OCP-HMACSHA1)) 2025-04-08 21:07:19.043 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=50b69f61-2806-4a02-a4c9-581269bae9ac, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000001, authType=OCP-HMACSHA1)) 2025-04-08 21:07:19.055 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:1000004 2025-04-08 21:07:19.083 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000016, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:19.085 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6bb7fdd25894631f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000012] 2025-04-08 21:07:20.009 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,78c8783909ba50ed] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000012, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:20.035 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,cb9612d7fff9d3bd] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:20.050 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,cb9612d7fff9d3bd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000012, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:20.008+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000012}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:20.066 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,cb9612d7fff9d3bd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000012, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:07:20.068 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,cb9612d7fff9d3bd] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000038] 2025-04-08 21:07:20.175 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:20.175 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:20.562 WARN 26087 --- [ocp-async-2,5bb95087fa1726b5,baa8ef33f9ec4761] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:20.562 INFO 26087 --- [ocp-async-10,6511aa9a333c55ba,5c9f10a730a90195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:07:21.035 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,82369993a0185d8b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000038, operation:EXECUTE, state: RUNNING 2025-04-08 21:07:21.078 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:07:21.084 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000038, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:07:21.033+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000038}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:07:21.087 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:07:21.097 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:07:21.105 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:07:21.107 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:07:21.111 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:07:21.119 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:07:21.128 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:07:21.138 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:21.145 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.151 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:21.158 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.199 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 21:07:21.227 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.229 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.233 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:21.238 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.280 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 21:07:21.312 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.315 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.324 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:07:21.333 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:07:21.343 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:07:21.355 INFO 26087 --- [prometheus-minute-schedule-1,727ef3aec81d710c,e62409a50179f3c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:21.384 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 21:07:21.407 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:07:21.410 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:07:21.411 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:07:21.413 ERROR 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:07:21.415 ERROR 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:07:21.418 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:07:21.423 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6dfb22c757278dcf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000038, operation:EXECUTE, state: FAILED 2025-04-08 21:07:21.805 INFO 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=1000006, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000001, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, createTime=2025-04-08T21:07:06+08:00, startTime=2025-04-08T21:07:06.093+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=1000010, name=Sync all tenant information, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=18, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000019, name=Refresh ocp agent config, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.RefreshAgentConfigTask, seriesId=15, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000008, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=28, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000038, name=Install ocp agent, state=FAILED, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask, seriesId=3, startTime=2025-04-08T21:07:21.034+08:00, endTime=2025-04-08T21:07:21.418+08:00}, SubtaskInstanceEntity{id=1000043, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=24, startTime=2025-04-08T21:07:08.820+08: 2025-04-08 21:07:21.810 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:07:21.811 INFO 26087 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000006, status:FAILED 2025-04-08 21:07:22.562 WARN 26087 --- [ocp-async-8,558ab6b5f38923cd,13fc2cfa6e5d58c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:22.562 INFO 26087 --- [ocp-async-10,0364fdc06b45e6bd,dd0a5b300a4d8f60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:23.176 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:23.176 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:24.355 INFO 26087 --- [prometheus-minute-schedule-1,72fce23f3fe9d4aa,383281d4abbd29cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:24.562 WARN 26087 --- [ocp-async-3,e6bea1743d8d1270,8976fcbf9a8a9267] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:24.562 INFO 26087 --- [ocp-async-7,15bf63a2ff43fc09,c824ac1d28cbb669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:26.176 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:26.177 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:26.525 INFO 26087 --- [ocp-schedules-27,3f8f8f3968af149d,984b6e91fcf9f5ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:07:26.525 INFO 26087 --- [ocp-schedules-12,25d4de4ceb9d0a8f,8f75011e2e9fe216] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:07:26.541 INFO 26087 --- [ocp-async-2,82f1d4c3c2c1a130,6653baeebd20dde1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:07:26.563 WARN 26087 --- [ocp-async-6,9d53b435b4d4caee,be08d49f42563ee7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:26.563 INFO 26087 --- [ocp-async-4,cfa3e6cd63030a09,7da447f315fe390d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:26.583 INFO 26087 --- [ocp-schedules-13,8d8864f3ebec8298,c8557915a1c414e1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:07:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:07:26.623480+08:00, lastProcessTime=2025-04-08T21:07:16.623537+08:00 2025-04-08 21:07:26.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:07:26.623480+08:00 2025-04-08 21:07:27.034 INFO 26087 --- [ocp-schedules-12,ef9fb2a56d0e55df,52e38df07698f41c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:07:27.035 INFO 26087 --- [ocp-schedules-12,ef9fb2a56d0e55df,52e38df07698f41c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:07:27.355 INFO 26087 --- [prometheus-minute-schedule-1,2162dab9d35c6849,016782fc61f469fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:27.740 INFO 26087 --- [ocp-schedules-10,b893aea63c24266c,3bd956e01f2bd1cc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:07:28.563 WARN 26087 --- [ocp-async-4,6cf56af5cc9827bf,91234bdb4786d7b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:28.563 INFO 26087 --- [ocp-async-2,ce0176928f520513,e4f493c1aef46b65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:29.177 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:29.178 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:30.355 INFO 26087 --- [prometheus-minute-schedule-1,981a601138d62b34,7924ee56563f4dcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:30.563 WARN 26087 --- [ocp-async-10,4a555e7b6a6b9cf2,2013fccb5586730d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:30.563 INFO 26087 --- [ocp-async-7,29f5d7603939e63e,82b93cc9aa24f9a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:31.182 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,6f881e09e421e688,a8bbfe737fd7d641] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=6f881e09e421e688, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:07:31.182 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,42fdde482fa7c17f,3b103126dfb74cee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=42fdde482fa7c17f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:07:31.182 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,6f881e09e421e688,a8bbfe737fd7d641] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=6f881e09e421e688, duration=3 ms] 2025-04-08 21:07:31.183 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,f92cab69eb4c7164,6f50189004edd17c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=f92cab69eb4c7164, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:07:31.183 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,f92cab69eb4c7164,6f50189004edd17c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=f92cab69eb4c7164, duration=4 ms] 2025-04-08 21:07:31.184 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,07558fbf2f349b9e,fddc0a625f9bc07c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=07558fbf2f349b9e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:07:31.186 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,42fdde482fa7c17f,3b103126dfb74cee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=42fdde482fa7c17f, duration=8 ms] 2025-04-08 21:07:31.272 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,07558fbf2f349b9e,fddc0a625f9bc07c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=07558fbf2f349b9e, duration=93 ms] 2025-04-08 21:07:31.280 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,4c3801883dfa2c50,bab5ed8ac03c7406] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=4c3801883dfa2c50, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:07:31.290 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,4c3801883dfa2c50,bab5ed8ac03c7406] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=4c3801883dfa2c50, duration=14 ms] 2025-04-08 21:07:31.319 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:07:31.319 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:07:31.319 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:07:31.325 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e543bc805bb35ea0,f056dc839a6e1d0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=e543bc805bb35ea0, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:07:31.341 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e543bc805bb35ea0,f056dc839a6e1d0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=e543bc805bb35ea0, duration=18 ms] 2025-04-08 21:07:31.919 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,fdac242ec816541e,d39f47d60b1bac34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments?taskInstanceId=1000006&subtaskInstanceId=1000038, client=10.11.12.102, traceId=fdac242ec816541e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000006,1000038,] 2025-04-08 21:07:31.923 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,fdac242ec816541e,d39f47d60b1bac34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments client=10.11.12.102, traceId=fdac242ec816541e, duration=9 ms] 2025-04-08 21:07:32.178 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:32.178 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:32.563 WARN 26087 --- [ocp-async-1,dda3af101ddbe4ee,3c3a075412988ed5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:32.563 INFO 26087 --- [ocp-async-7,74d42aedc18a7af7,8a7e1929dd6a51e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:33.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,015d1ad78915bc0f,633f3e07d64971cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=015d1ad78915bc0f, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:07:33.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,015d1ad78915bc0f,633f3e07d64971cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=015d1ad78915bc0f, duration=2 ms] 2025-04-08 21:07:33.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,418e0b6489bb0525,15c8788991e185c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=418e0b6489bb0525, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:07:33.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,bda1af304a7a927b,46e9184a7f1fe60d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=bda1af304a7a927b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:07:33.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,418e0b6489bb0525,15c8788991e185c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=418e0b6489bb0525, duration=3 ms] 2025-04-08 21:07:33.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,bda1af304a7a927b,46e9184a7f1fe60d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=bda1af304a7a927b, duration=3 ms] 2025-04-08 21:07:33.204 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:07:33.204 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:07:33.204 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:07:33.355 INFO 26087 --- [prometheus-minute-schedule-1,98fb7f3928aec049,c4f3434b751a93bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:34.563 WARN 26087 --- [ocp-async-7,fd16c54f8001fa72,91e8d21925ace30e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:34.563 INFO 26087 --- [ocp-async-4,434094529f39e91f,b38ae59037db1502] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:07:34.995 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:07:34.996 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:07:34.996 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:07:35.001 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000001), status=TAKINGOVER 2025-04-08 21:07:35.001 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 21:07:35.179 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:35.179 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:36.175 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,cd90875f65734dbd,5368c8ce78252dd5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=cd90875f65734dbd, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:07:36.177 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,cd90875f65734dbd,5368c8ce78252dd5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=cd90875f65734dbd, duration=5 ms] 2025-04-08 21:07:36.355 INFO 26087 --- [prometheus-minute-schedule-1,87d2a720302b6deb,554518f73349c9e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:36.541 INFO 26087 --- [ocp-async-4,7c69db7dc0c938b3,e885044697f0e952] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:07:36.556 INFO 26087 --- [ocp-schedules-20,02a0a4e5f13bafce,c5746f318a13f0f9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:07:36.557 INFO 26087 --- [ocp-schedules-20,02a0a4e5f13bafce,c5746f318a13f0f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:36.557 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:07:36.563 WARN 26087 --- [ocp-async-6,6026b42eeb88ca0e,7253f39f19943ed0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:36.563 INFO 26087 --- [ocp-async-10,f529affac4798827,57c7b7e3573f5d87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:36.584 INFO 26087 --- [ocp-schedules-22,e07411ac0ea54ee6,b549ce68a18634ee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:07:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:07:36.623661+08:00, lastProcessTime=2025-04-08T21:07:26.623480+08:00 2025-04-08 21:07:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:07:36.623661+08:00 2025-04-08 21:07:36.793 INFO 26087 --- [ocp-schedules-5,85a49249d3f473b9,a230881ca1748543] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:07:36.801 INFO 26087 --- [ocp-schedules-5,85a49249d3f473b9,a230881ca1748543] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:07:36.801 INFO 26087 --- [ocp-schedules-5,85a49249d3f473b9,a230881ca1748543] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:07:37.040 INFO 26087 --- [ocp-schedules-9,b4f2c7a6fa92d999,3185a3bb327c6e60] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:07:37.041 INFO 26087 --- [ocp-schedules-9,b4f2c7a6fa92d999,3185a3bb327c6e60] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:07:37.766 INFO 26087 --- [ocp-schedules-28,a6aa80e98648b317,5de307376112f36b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:07:38.179 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:38.180 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:38.563 WARN 26087 --- [ocp-async-8,3b34cbf28afe766b,4f9f9a740f08ab53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:38.563 INFO 26087 --- [ocp-async-10,255c05f63658797a,60aa75f93ad59011] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:39.355 INFO 26087 --- [prometheus-minute-schedule-1,19eb65e7de2efb37,c916aeebf04db84d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:40.001 INFO 26087 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:07:40.001 INFO 26087 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:07:40.564 WARN 26087 --- [ocp-async-10,a57b83bf040e3a7a,78d2062deefe998a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:40.564 INFO 26087 --- [ocp-async-3,6366ee840b40ac69,81d63dca100369fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:41.180 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:41.181 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:42.355 INFO 26087 --- [prometheus-minute-schedule-1,345c4593aa2c8351,37d030623e8c7082] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:42.564 WARN 26087 --- [ocp-async-4,e30140a564369e53,5248a631255ee28e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:42.564 INFO 26087 --- [ocp-async-1,ffb0f0118ed12896,1f0c605931025c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:44.181 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:44.182 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:44.564 WARN 26087 --- [ocp-async-7,3b069b640f90ba5e,8eb9ec7d08cf5667] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:44.564 INFO 26087 --- [ocp-async-8,586e6465cccc2018,f9d79d5374975b73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:45.355 INFO 26087 --- [prometheus-minute-schedule-1,711b8dc82fbc56b4,6dfa022a67aa86d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:46.521 INFO 26087 --- [ocp-schedules-13,ba3135cdfac417c8,480de69b52eb2e46] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:07:46.522 INFO 26087 --- [ocp-schedules-18,4f015ff8262a42ae,351f8fdf809b5670] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:07:46.522 INFO 26087 --- [ocp-schedules-18,4f015ff8262a42ae,351f8fdf809b5670] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:07:46.523 INFO 26087 --- [ocp-schedules-12,3666af7e03768921,286c5bc6cb273da6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:07:46.523 INFO 26087 --- [ocp-schedules-5,0b6c725f2707ed40,99331ce51e759b4a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:07:46.523 INFO 26087 --- [ocp-schedules-12,3666af7e03768921,286c5bc6cb273da6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:07:46.523 INFO 26087 --- [ocp-schedules-5,0b6c725f2707ed40,99331ce51e759b4a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:07:46.524 INFO 26087 --- [ocp-schedules-11,e9b7b97028b59a36,9e0b769d6a1d5a22] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:07:46.524 INFO 26087 --- [ocp-schedules-21,c070fe144c33901f,d2621e0971b0f215] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:07:46.525 INFO 26087 --- [ocp-schedules-11,e9b7b97028b59a36,9e0b769d6a1d5a22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:46.525 INFO 26087 --- [ocp-schedules-19,c370e34f8290a50a,cbeadc3afedfee39] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:07:46.525 INFO 26087 --- [ocp-schedules-21,c070fe144c33901f,d2621e0971b0f215] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:07:46.530 INFO 26087 --- [ocp-schedules-16,c224ec4d18c503da,3c8bc6041f1a0752] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:07:46.530 INFO 26087 --- [ocp-schedules-16,8ffd1365787bf4f4,3574fbf758eff313] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:07:46.531 INFO 26087 --- [ocp-schedules-16,8ffd1365787bf4f4,3574fbf758eff313] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:07:46.536 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000001, status=TAKINGOVER 2025-04-08 21:07:46.536 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:07:46.541 INFO 26087 --- [ocp-async-8,736ba25a76f11972,9d0bfc9f8cd80c41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:07:46.558 INFO 26087 --- [ocp-schedules-20,02a0a4e5f13bafce,c5746f318a13f0f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:07:46.564 WARN 26087 --- [ocp-async-3,52474fb6e4057662,943a6fe5316f96a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:46.564 INFO 26087 --- [ocp-async-10,c9ceec5c8204b1bb,23b6d6a87af4f80a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:46.581 INFO 26087 --- [ocp-schedules-4,8d7120884487c303,37ab4524872480be] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:07:46.585 INFO 26087 --- [ocp-schedules-15,726259f1682cd83a,e7d4a8e5d0f5aec7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:07:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:07:46.623600+08:00, lastProcessTime=2025-04-08T21:07:36.623661+08:00 2025-04-08 21:07:46.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:07:46.623600+08:00 2025-04-08 21:07:47.044 INFO 26087 --- [ocp-schedules-17,4907f29cbe64c43b,724358a52bcd7d29] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:07:47.046 INFO 26087 --- [ocp-schedules-17,4907f29cbe64c43b,724358a52bcd7d29] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:07:47.182 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:47.182 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:47.795 INFO 26087 --- [ocp-schedules-22,2244155a1b38ecf2,372f49bb4628c547] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:07:48.355 INFO 26087 --- [prometheus-minute-schedule-1,c51a43fb9fc7fc37,33215c115a5d48f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:48.564 INFO 26087 --- [ocp-async-6,573c6709e890a3ef,c626bc4567883e34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:48.564 WARN 26087 --- [ocp-async-9,ea8efa251eae4809,93df91c9420fcbe2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:50.183 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:50.183 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:50.564 WARN 26087 --- [ocp-async-1,e3b30d9a42c69a9e,ae90f3c2e26611af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:50.564 INFO 26087 --- [ocp-async-7,15c41a1eef30764c,3314d26c42ec00ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:51.355 INFO 26087 --- [prometheus-minute-schedule-1,6736dfcfe85cc947,e75b75745f42f57c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:52.564 WARN 26087 --- [ocp-async-8,a225b674fda55964,d9038cddd5dfa27a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:52.565 INFO 26087 --- [ocp-async-3,9b84facf1cd1e649,39536f56e04c71f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:53.183 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:53.184 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:54.355 INFO 26087 --- [prometheus-minute-schedule-1,764ad1e3fb3eaaac,d340fff7875d8c5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:54.565 WARN 26087 --- [ocp-async-10,9fda31b5f7a2da2c,591116d926269675] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:54.565 INFO 26087 --- [ocp-async-4,eb0e5ee2edadc269,6056ffcd1bce21f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:56.184 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:56.185 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:07:56.525 INFO 26087 --- [ocp-schedules-11,e9b7b97028b59a36,9e0b769d6a1d5a22] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:07:56.526 INFO 26087 --- [ocp-schedules-21,c070fe144c33901f,d2621e0971b0f215] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:07:56.542 INFO 26087 --- [ocp-async-4,3992224d7a1ac9ae,d17506fa2b51e505] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:07:56.565 WARN 26087 --- [ocp-async-7,9b0bd7f7c9e11874,35f703b92e74d418] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:56.565 INFO 26087 --- [ocp-async-5,867e9ec5748fdb6c,5b49cc98f44825b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:56.586 INFO 26087 --- [ocp-schedules-17,6dd6b9bf5ed9700f,865ad348fa7151e5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:07:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:07:56.623813+08:00, lastProcessTime=2025-04-08T21:07:46.623600+08:00 2025-04-08 21:07:56.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:07:56.623813+08:00 2025-04-08 21:07:56.802 INFO 26087 --- [ocp-schedules-8,8de4d04adcdbca02,f72ce6d2726a7337] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:07:56.812 INFO 26087 --- [ocp-schedules-8,8de4d04adcdbca02,f72ce6d2726a7337] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:07:56.812 INFO 26087 --- [ocp-schedules-8,8de4d04adcdbca02,f72ce6d2726a7337] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:07:57.054 INFO 26087 --- [ocp-schedules-27,49c3cbb8f78db4f1,ec75575db3602586] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:07:57.055 INFO 26087 --- [ocp-schedules-27,49c3cbb8f78db4f1,ec75575db3602586] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:07:57.355 INFO 26087 --- [prometheus-minute-schedule-1,7466250eff64ec7a,d048b2710e0fbdd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117680, interval=60 2025-04-08 21:07:57.820 INFO 26087 --- [ocp-schedules-6,1ffc4a4b0581a8fb,e192b74a682ae9e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:07:58.565 WARN 26087 --- [ocp-async-3,07f71a97bbf976cc,cc6c430aa560eb30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:07:58.565 INFO 26087 --- [ocp-async-2,cb39282e7cbd8a2b,f1745ef141c65bdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:07:59.185 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:07:59.185 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:00.355 INFO 26087 --- [prometheus-minute-schedule-1,433801afe03814f2,08b79e3e7e370b81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:00.565 WARN 26087 --- [ocp-async-6,2671562305d97f2e,875a6a14176e9bcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:00.565 INFO 26087 --- [ocp-async-10,e3003f924890e619,f3b99d7394bdefe5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:02.185 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:02.186 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:02.565 WARN 26087 --- [ocp-async-7,34e41aec3b6f92c0,8e4ec915e26ebfa4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:02.565 INFO 26087 --- [ocp-async-4,1e5f7d9fb4b82343,9f2e1927aaace31a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:03.355 INFO 26087 --- [prometheus-minute-schedule-1,b3833838db24c38d,a35d64c735cd1cb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:04.565 WARN 26087 --- [ocp-async-8,3460c46842e12f92,4b2dfe6ff163467a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:04.565 INFO 26087 --- [ocp-async-5,0be04779b49ffbc6,b93548fe15a7dab6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:05.186 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:05.187 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:06.262 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:04:06.208, end=2025-04-08T21:06:06.259} 2025-04-08 21:08:06.265 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:08:06.266 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:08:06.355 INFO 26087 --- [prometheus-minute-schedule-1,b1b50d02ae95272a,d82b0fa72cfd84a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:06.355 INFO 26087 --- [prometheus-cache-monitor1,e4e07deffc74d80d,2a9a7866a3b197d8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":730,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":366} 2025-04-08 21:08:06.355 INFO 26087 --- [prometheus-cache-monitor1,e4e07deffc74d80d,2a9a7866a3b197d8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:08:06.542 INFO 26087 --- [ocp-async-2,97b921841a010549,3f5329f899321142] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:08:06.566 WARN 26087 --- [ocp-async-10,365f6bfc3871083e,7fbbe613c39c84a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:06.566 INFO 26087 --- [ocp-async-6,f47be898358eb2f2,16e31b13f867f5a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:06.587 INFO 26087 --- [ocp-schedules-28,7765dba2d2094f5f,1d2bede457863824] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:08:06.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:08:06.624980+08:00, lastProcessTime=2025-04-08T21:07:56.623813+08:00 2025-04-08 21:08:06.731 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:08:06.624980+08:00 2025-04-08 21:08:07.061 INFO 26087 --- [ocp-schedules-6,93715f3cc3c0b6c4,e58d9400c19217ae] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:08:07.062 INFO 26087 --- [ocp-schedules-6,93715f3cc3c0b6c4,e58d9400c19217ae] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:08:07.855 INFO 26087 --- [ocp-schedules-19,553278ff51acd0b2,e90e84557e2c7edf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:08:08.187 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:08.188 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:08.566 WARN 26087 --- [ocp-async-1,eb3ff8fd2dd36465,30c7e6aadfc53183] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:08.566 INFO 26087 --- [ocp-async-4,e8a1b760e74dca16,64870754e438654d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:09.355 INFO 26087 --- [prometheus-minute-schedule-1,0e9753e1b89f8924,ad66308ed1ec0f65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:10.002 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:08:10.002 INFO 26087 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:08:10.566 WARN 26087 --- [ocp-async-3,fffa4777fbf0005b,c66bdf485c1e9fef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:10.566 INFO 26087 --- [ocp-async-4,0a03edd9b2fe3fa5,0851588cacb51d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:08:11.188 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:11.188 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:12.343 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117620, nodeCount=374, affectRows=374 2025-04-08 21:08:12.345 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117620, nodeCount=371, validCount=371 2025-04-08 21:08:12.355 INFO 26087 --- [prometheus-minute-schedule-1,36e27a2244116c94,86678efcf26751a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:12.566 WARN 26087 --- [ocp-async-4,83f5ba747def1823,ca37d1d28dad38d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:12.566 INFO 26087 --- [ocp-async-10,afae2c918b5f32a4,97b9dd0fc278de69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:14.189 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:14.189 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:14.566 WARN 26087 --- [ocp-async-1,56a275e14b6663a2,e69d870082c83049] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:14.566 INFO 26087 --- [ocp-async-10,f14b18725cc044a5,4e9119fbe1a30985] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:15.355 INFO 26087 --- [prometheus-minute-schedule-1,6e3b0f310a7594e8,ac7c424eb62ac8d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:16.517 INFO 26087 --- [ocp-schedules-22,1f166963fb32340b,bf4da96d6d60ff62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:08:16.518 INFO 26087 --- [ocp-schedules-17,f077b4fefea3f866,4b79f8935cee8e29] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:08:16.521 INFO 26087 --- [ocp-schedules-9,94e0baf40357b322,469689c40ba435e9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:08:16.521 INFO 26087 --- [ocp-schedules-9,94e0baf40357b322,469689c40ba435e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:08:16.521 INFO 26087 --- [ocp-schedules-12,7a3e658c826457f8,ef8044e08b08a56e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:08:16.523 INFO 26087 --- [ocp-schedules-4,cdc4e9bdfae033cc,c72f3f168e36fdfa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:08:16.523 INFO 26087 --- [ocp-schedules-4,cdc4e9bdfae033cc,c72f3f168e36fdfa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:08:16.523 INFO 26087 --- [ocp-schedules-7,83015efc46e6ba2d,6ac3ac6d5e715af7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:08:16.523 INFO 26087 --- [ocp-schedules-7,83015efc46e6ba2d,6ac3ac6d5e715af7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:08:16.524 INFO 26087 --- [ocp-schedules-17,f077b4fefea3f866,4b79f8935cee8e29] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:08:16.524 INFO 26087 --- [ocp-schedules-17,747599377cfd8d5e,d890ed70c2fab621] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:08:16.524 INFO 26087 --- [ocp-schedules-19,24a2a650c6e8f42d,df2a0f0461ed98f3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:08:16.525 INFO 26087 --- [ocp-schedules-17,747599377cfd8d5e,d890ed70c2fab621] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:16.525 INFO 26087 --- [ocp-schedules-19,24a2a650c6e8f42d,df2a0f0461ed98f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:08:16.537 INFO 26087 --- [ocp-schedules-16,8ffd1365787bf4f4,3574fbf758eff313] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:08:16.537 INFO 26087 --- [ocp-schedules-16,88f3726b6e1ceb2a,6a8f013e0a9d7c9b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:08:16.537 INFO 26087 --- [ocp-schedules-16,88f3726b6e1ceb2a,6a8f013e0a9d7c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:16.540 INFO 26087 --- [ocp-schedules-22,1f166963fb32340b,bf4da96d6d60ff62] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:08:16.542 INFO 26087 --- [ocp-async-5,30522bc2af3520c4,ff677706abc4052c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:08:16.543 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000001, status=TAKINGOVER 2025-04-08 21:08:16.543 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:08:16.545 INFO 26087 --- [ocp-schedules-15,b553e05ebe1c15c1,a46c3fd87e4e91b6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:08:16.545 INFO 26087 --- [ocp-schedules-15,b553e05ebe1c15c1,a46c3fd87e4e91b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:16.546 INFO 26087 --- [ocp-schedules-9,05c169adb8bfc5c7,86993610bb5fa662] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:08:16.547 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:08:16.547 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:08:16.551 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:08:16.551 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:08:16.551 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:08:16.553 INFO 26087 --- [pool-overview-container-8,0f33f64143e8e363,f17ae0e2f2b8a10b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:16.553 INFO 26087 --- [pool-overview-container-4,4859620bfb4f69d2,006222622dd5ce3a] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:16.553 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:08:16.553 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:08:16.553 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:08:16.557 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:08:16.557 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:08:16.557 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:08:16.559 INFO 26087 --- [pool-11-thread-8,4859620bfb4f69d2,21be9378f4d3aea6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:08:16.559 INFO 26087 --- [pool-11-thread-6,4859620bfb4f69d2,d90b0797020b8cfc] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:08:16.560 INFO 26087 --- [pool-11-thread-7,4859620bfb4f69d2,68eef116af3c0f21] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:08:16.560 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:08:16.560 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2025-04-08 21:08:16.560 INFO 26087 --- [pool-11-thread-5,4859620bfb4f69d2,426116d9569bce55] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:08:16.560 INFO 26087 --- [pool-11-thread-5,4859620bfb4f69d2,426116d9569bce55] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:08:16.561 INFO 26087 --- [ocp-schedules-20,e2de3ad16341cbad,560adb23fb3be321] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:08:16.562 INFO 26087 --- [ForkJoinPool-7-worker-11,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:16.562 INFO 26087 --- [ForkJoinPool-7-worker-4,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:16.562 INFO 26087 --- [pool-11-thread-6,4859620bfb4f69d2,d90b0797020b8cfc] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:08:16.562 INFO 26087 --- [pool-11-thread-6,4859620bfb4f69d2,d90b0797020b8cfc] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:08:16.562 INFO 26087 --- [ForkJoinPool-7-worker-11,4859620bfb4f69d2,4174f99775a03c07] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000001, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:08:16.562 INFO 26087 --- [pool-11-thread-8,4859620bfb4f69d2,21be9378f4d3aea6] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:08:16.562 INFO 26087 --- [ForkJoinPool-7-worker-4,4859620bfb4f69d2,fd7b8449e7ec506c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 21:08:16.562 INFO 26087 --- [ForkJoinPool-7-worker-4,4859620bfb4f69d2,fd7b8449e7ec506c] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000001, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:08:16.562 INFO 26087 --- [ForkJoinPool-7-worker-4,4859620bfb4f69d2,fd7b8449e7ec506c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 21:08:16.562 INFO 26087 --- [pool-11-thread-7,4859620bfb4f69d2,68eef116af3c0f21] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:08:16.563 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:08:16.563 INFO 26087 --- [pool-11-thread-5,4859620bfb4f69d2,426116d9569bce55] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:08:16.564 INFO 26087 --- [pool-11-thread-5,4859620bfb4f69d2,426116d9569bce55] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:08:16.565 WARN 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:08:16.566 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:08:16.566 WARN 26087 --- [ocp-async-8,4536f387f40f8938,d28a553130b1334c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:16.567 INFO 26087 --- [ocp-async-4,8497ad40c3c923aa,74000f53407a6cb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:16.567 WARN 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:08:16.573 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.e.internal.connector.Connectors : init global executor service. 2025-04-08 21:08:16.577 INFO 26087 --- [ocp-schedules-7,c87d0a55ef83d889,0154b2ca5a49cee6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:08:16.582 INFO 26087 --- [ocp-schedules-2,a7d690858ae14a12,230d770dd83b332d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:08:16.587 INFO 26087 --- [ocp-schedules-2,ea02890ef1127e04,bae761bffed1b106] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:08:16.587+08:00, clusters=[1000001] 2025-04-08 21:08:16.588 INFO 26087 --- [ocp-schedules-6,f239d516be9aacb9,e77c486191a702c4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:08:16.588+08:00, clusters=[1000001] 2025-04-08 21:08:16.588 INFO 26087 --- [ocp-schedules-6,f239d516be9aacb9,e77c486191a702c4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:08:16.588 INFO 26087 --- [ocp-schedules-5,bf6daf17a40b2863,a13bef4c4ab2be09] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:08:16.589 INFO 26087 --- [ocp-schedules-21,18985971c6ac27f5,2f7c5c069cbd413c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:08:16.589 INFO 26087 --- [ocp-schedules-2,ea02890ef1127e04,bae761bffed1b106] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:08:16.598 INFO 26087 --- [ocp-schedules-10,a7927affc9b46fa1,ab3775b162ee0bfe] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:08:16.598 INFO 26087 --- [ocp-schedules-10,a7927affc9b46fa1,ab3775b162ee0bfe] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:08:16.612 INFO 26087 --- [ocp-schedules-24,31dc31595477ae60,4dd8012afa299c31] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:08:16.612 INFO 26087 --- [ocp-schedules-24,31dc31595477ae60,4dd8012afa299c31] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:08:16.617 INFO 26087 --- [ocp-schedules-24,31dc31595477ae60,4dd8012afa299c31] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:08:16.618 INFO 26087 --- [ocp-schedules-24,31dc31595477ae60,4dd8012afa299c31] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:08:16.622 INFO 26087 --- [ocp-schedules-30,e4cd645116eebd79,e63a62b7beb508c9] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:08:16.622 INFO 26087 --- [ocp-schedules-30,e4cd645116eebd79,e63a62b7beb508c9] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:08:16.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:08:16.624122+08:00, lastProcessTime=2025-04-08T21:08:06.624980+08:00 2025-04-08 21:08:16.630 INFO 26087 --- [ocp-schedules-29,20b54796edbb09d1,a3e633430f19ed2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:08:16.634 INFO 26087 --- [ocp-schedules-26,9faea6fa96009515,2a792d0dea351234] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:08:16.649 INFO 26087 --- [ocp-schedules-28,8201cbd9720a64f0,89e6d49bdb898814] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:08:16.650 INFO 26087 --- [ocp-schedules-28,8201cbd9720a64f0,89e6d49bdb898814] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:16.652 INFO 26087 --- [ocp-schedules-23,30600c1201dd6727,24adef70c1407bc0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:08:16.655 INFO 26087 --- [ocp-schedules-23,30600c1201dd6727,24adef70c1407bc0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:08:16.655 INFO 26087 --- [ocp-schedules-23,30600c1201dd6727,24adef70c1407bc0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:08:16.666 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:08:16.667 ERROR 26087 --- [ocp-schedules-28,8201cbd9720a64f0,89e6d49bdb898814] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:08:16.668 INFO 26087 --- [ocp-schedules-28,8201cbd9720a64f0,89e6d49bdb898814] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:08:16.696 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.61, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:3 2025-04-08 21:08:16.696 INFO 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:08:16.731 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:08:16.624122+08:00 2025-04-08 21:08:16.795 INFO 26087 --- [ocp-schedules-8,6ac56bb681fe1289,8dc28d844db5b68d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:08:16.795 INFO 26087 --- [ocp-schedules-8,6ac56bb681fe1289,8dc28d844db5b68d] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:08:16.801 INFO 26087 --- [pool-23-thread-1,6ac56bb681fe1289,267e1cfc8d761ca5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:08:16.803 INFO 26087 --- [ocp-schedules-20,7105e4fb4ece0487,6d6d50623b236aa9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:08:16.803 INFO 26087 --- [ocp-schedules-20,7105e4fb4ece0487,6d6d50623b236aa9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:08:16.803 INFO 26087 --- [ocp-schedules-20,7105e4fb4ece0487,6d6d50623b236aa9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:08:16.803 INFO 26087 --- [ocp-schedules-20,7105e4fb4ece0487,6d6d50623b236aa9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:08:16.803 INFO 26087 --- [ocp-schedules-20,7105e4fb4ece0487,6d6d50623b236aa9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:08:16.803 INFO 26087 --- [ocp-schedules-20,7105e4fb4ece0487,6d6d50623b236aa9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:08:16.807 WARN 26087 --- [ocp-schedules-8,6ac56bb681fe1289,8dc28d844db5b68d] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$14662e73.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:08:16.807 INFO 26087 --- [ocp-schedules-8,6ac56bb681fe1289,8dc28d844db5b68d] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:08:16.813 INFO 26087 --- [ocp-schedules-22,cac3e1a7bb7bcbec,6264f0cf1b99278c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:08:16.828 INFO 26087 --- [ocp-schedules-22,cac3e1a7bb7bcbec,6264f0cf1b99278c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:08:16.828 INFO 26087 --- [ocp-schedules-22,cac3e1a7bb7bcbec,6264f0cf1b99278c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:08:17.066 INFO 26087 --- [ocp-schedules-1,4a09fbc5b4b5483b,c8b080fd576991a1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:08:17.067 INFO 26087 --- [ocp-schedules-1,4a09fbc5b4b5483b,c8b080fd576991a1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:08:17.110 INFO 26087 --- [ocp-schedules-5,baa316de624e0094,d558db3e7f9148d2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:08:17.118 INFO 26087 --- [ocp-schedules-5,baa316de624e0094,d558db3e7f9148d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:08:17.119 WARN 26087 --- [ocp-schedules-5,baa316de624e0094,d558db3e7f9148d2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:08:17.120 INFO 26087 --- [ocp-schedules-5,baa316de624e0094,d558db3e7f9148d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:08:17.121 WARN 26087 --- [ocp-schedules-5,baa316de624e0094,d558db3e7f9148d2] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:08:17.123 INFO 26087 --- [ocp-schedules-5,baa316de624e0094,d558db3e7f9148d2] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=2 2025-04-08 21:08:17.189 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:17.190 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:17.364 WARN 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:08:17.365 ERROR 26087 --- [ocp-schedules-18,e5b824160eaea8ba,9abf438ce5d8c6ed] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 21:08:17.880 INFO 26087 --- [ocp-schedules-11,18d7e5b305cee8c9,283eed262bd693da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:08:18.355 INFO 26087 --- [prometheus-minute-schedule-1,6762c5d80d4ad60d,f24d7a38734612b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:18.567 WARN 26087 --- [ocp-async-4,e4beb6a3c63e1140,da97f8781a617717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:18.567 INFO 26087 --- [ocp-async-1,4c6f82e1a4534990,f92ed6dec792b21e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:20.190 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:20.190 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:20.567 WARN 26087 --- [ocp-async-1,7d89fb9ecd631f87,acc927d52ab1dd97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:20.567 INFO 26087 --- [ocp-async-10,f2fe745a5a02af6c,030f16875d3a78b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:08:21.355 INFO 26087 --- [prometheus-minute-schedule-1,167ecf313ed5718e,72b10d4d0f6a85bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:22.567 WARN 26087 --- [ocp-async-10,2adca9329fc2ddcd,479178667a497a6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:22.567 INFO 26087 --- [ocp-async-5,633c98e44bfc3815,10ecd4bfc56d3087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:23.191 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:23.191 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:24.355 INFO 26087 --- [prometheus-minute-schedule-1,251e2ffd5d277bbe,311fa25eb098d22d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:24.567 WARN 26087 --- [ocp-async-6,8ee3d0ad992ebe8b,dc994f9040a5d1b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:24.567 INFO 26087 --- [ocp-async-5,2abe9a9df8d1dc48,cb79019cef3b31c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:26.191 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:26.192 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:26.525 INFO 26087 --- [ocp-schedules-19,24a2a650c6e8f42d,df2a0f0461ed98f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:08:26.525 INFO 26087 --- [ocp-schedules-17,747599377cfd8d5e,d890ed70c2fab621] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:08:26.542 INFO 26087 --- [ocp-async-1,81ce7f6d66c144c6,7d05d6fa2f1ef4ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:08:26.546 INFO 26087 --- [ocp-schedules-15,b553e05ebe1c15c1,a46c3fd87e4e91b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 21:08:26.567 WARN 26087 --- [ocp-async-10,8d8ed2d32d008d86,06e7eeeee2fad0b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:26.567 INFO 26087 --- [ocp-async-3,eacf417a8012a749,f342cfa4e0adf3f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:26.590 INFO 26087 --- [ocp-schedules-22,7833e648b6a0d53c,a6b709ea2c011f63] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:08:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:08:26.623695+08:00, lastProcessTime=2025-04-08T21:08:16.624122+08:00 2025-04-08 21:08:26.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:08:26.623695+08:00 2025-04-08 21:08:27.076 INFO 26087 --- [ocp-schedules-19,5ec33150344025d6,b7709fa28250384c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:08:27.077 INFO 26087 --- [ocp-schedules-19,5ec33150344025d6,b7709fa28250384c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:08:27.355 INFO 26087 --- [prometheus-minute-schedule-1,6a6f11425f0d7678,f11b634b90c8f515] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:27.916 INFO 26087 --- [ocp-schedules-3,05b04b09290c02d2,a66bc0486ebc11e2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:08:28.567 WARN 26087 --- [ocp-async-6,6843219ea4e6c283,4617af6fc04bb99f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:28.567 INFO 26087 --- [ocp-async-3,457d713e91a7763d,887cb45bb88854d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:29.192 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:29.193 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:30.355 INFO 26087 --- [prometheus-minute-schedule-1,142037e284ed8acb,9bb9be6f925a4ef7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:30.567 WARN 26087 --- [ocp-async-5,4258ef83f20617f7,0c56e85a4f5b1827] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:30.568 INFO 26087 --- [ocp-async-7,89ecb50f8f4a6172,de7133f8e648fb7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:32.193 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:32.193 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:32.568 WARN 26087 --- [ocp-async-7,08ee38a7ac0276e3,dabe7b93f73d29c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:32.568 INFO 26087 --- [ocp-async-9,f615e7192613dcf0,53f0332d55501257] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:33.355 INFO 26087 --- [prometheus-minute-schedule-1,06a527b5f9631e47,e4748f6852fe6e84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:33.428 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,4bb11ab4e2589ee1,4beaaef87e9298c6] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/1000006/rollback, client 10.11.12.102, traceId 4bb11ab4e2589ee1] 2025-04-08 21:08:33.431 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,4bb11ab4e2589ee1,4beaaef87e9298c6] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/1000006/rollback?taskInstanceId=1000006, client=10.11.12.102, traceId=4bb11ab4e2589ee1, method=NoDataResponse com.oceanbase.ocp.server.common.controller.TaskController.rollbackTask(Long), args=1000006,] 2025-04-08 21:08:33.437 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,4bb11ab4e2589ee1,4beaaef87e9298c6] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 1000038, current state: FAILED, new state: READY 2025-04-08 21:08:33.447 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,4bb11ab4e2589ee1,4beaaef87e9298c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/1000006/rollback client=10.11.12.102, traceId=4bb11ab4e2589ee1, duration=21 ms] 2025-04-08 21:08:33.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,d0493640a0e7fd5e,008da3b0817360ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=d0493640a0e7fd5e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:33.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,37a9ad0be396204c,8e6623cc18694701] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=37a9ad0be396204c, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:08:33.619 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,37a9ad0be396204c,8e6623cc18694701] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=37a9ad0be396204c, duration=4 ms] 2025-04-08 21:08:33.629 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,d0493640a0e7fd5e,008da3b0817360ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=d0493640a0e7fd5e, duration=14 ms] 2025-04-08 21:08:34.143 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,33db954b3cc55bd8,1b4c4bce80096f6d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments?taskInstanceId=1000006&subtaskInstanceId=1000038, client=10.11.12.102, traceId=33db954b3cc55bd8, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000006,1000038,] 2025-04-08 21:08:34.146 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,33db954b3cc55bd8,1b4c4bce80096f6d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments client=10.11.12.102, traceId=33db954b3cc55bd8, duration=7 ms] 2025-04-08 21:08:34.154 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,9cc3d6b73486cda3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000038, original=READY 2025-04-08 21:08:34.156 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,9cc3d6b73486cda3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000038, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:34.171 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:34.174 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000038, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:34.154+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000038}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:34.177 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:08:34.179 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:08:34.185 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:08:34.207 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:08:34.221 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.263 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:08:34.267 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.271 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:08:34.290 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.299 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.331 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:08:34.337 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.340 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:08:34.353 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.356 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.393 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:08:34.402 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.405 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:08:34.428 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.431 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.432 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:08:34.469 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:08:34.474 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.480 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:08:34.568 WARN 26087 --- [ocp-async-2,c05689c7bd81aee7,880e9080558243fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:34.568 INFO 26087 --- [ocp-async-1,cf302c64962b3fef,835246aa9b9897c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:34.827 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.846 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.853 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:08:34.857 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:08:34.860 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:08:34.882 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:08:34.889 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:34.904 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 21:08:35.009 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,930d425c9effd93e,120942ecab94d8c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=930d425c9effd93e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:35.019 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,930d425c9effd93e,120942ecab94d8c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=930d425c9effd93e, duration=15 ms] 2025-04-08 21:08:35.193 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:35.194 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:35.237 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,57db8a9d950c5d14,d4dcf41ecd0f6964] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments?taskInstanceId=1000006&subtaskInstanceId=1000038, client=10.11.12.102, traceId=57db8a9d950c5d14, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000006,1000038,] 2025-04-08 21:08:35.239 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,57db8a9d950c5d14,d4dcf41ecd0f6964] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments client=10.11.12.102, traceId=57db8a9d950c5d14, duration=6 ms] 2025-04-08 21:08:35.330 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:35.335 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=admin, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:08:35.340 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:08:35.345 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:08:35.348 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:08:35.356 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000038, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:35.357 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,9010804c2a4c9471] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000012] 2025-04-08 21:08:36.168 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4915c0347f3bc43d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000012, original=READY 2025-04-08 21:08:36.169 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4915c0347f3bc43d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000012, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:36.182 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,672ea15f7347089e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:36.202 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,672ea15f7347089e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000012, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:36.168+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000012}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:36.215 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,672ea15f7347089e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000012, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:36.216 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,672ea15f7347089e] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000016] 2025-04-08 21:08:36.228 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,bb0bdcdf97a6c370,cb46b0e9174abc0d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=bb0bdcdf97a6c370, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:36.238 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,bb0bdcdf97a6c370,cb46b0e9174abc0d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=bb0bdcdf97a6c370, duration=15 ms] 2025-04-08 21:08:36.355 INFO 26087 --- [prometheus-minute-schedule-1,ded5e1d5e2d276f1,6ad6d04027aea11a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:36.425 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,3344c401c8cb4625,79a9cd87e6fb5c79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments?taskInstanceId=1000006&subtaskInstanceId=1000038, client=10.11.12.102, traceId=3344c401c8cb4625, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000006,1000038,] 2025-04-08 21:08:36.427 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,3344c401c8cb4625,79a9cd87e6fb5c79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006/subtasks/1000038/logSegments client=10.11.12.102, traceId=3344c401c8cb4625, duration=6 ms] 2025-04-08 21:08:36.542 INFO 26087 --- [ocp-async-3,4c4b7e835b0f2778,1cd1e5b6649fadf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:08:36.568 WARN 26087 --- [ocp-async-5,b7a8fc3de6c9b471,eb31197f152d9413] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:36.568 INFO 26087 --- [ocp-async-10,98044dfb5fd2f728,cf738835a87aa202] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:36.591 INFO 26087 --- [ocp-schedules-19,cdaaaf92bf926fef,65167e8638fa5972] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:08:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:08:36.623537+08:00, lastProcessTime=2025-04-08T21:08:26.623695+08:00 2025-04-08 21:08:36.733 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:08:36.623537+08:00 2025-04-08 21:08:36.829 INFO 26087 --- [ocp-schedules-13,27363fc59c874e08,d9ad0a1565e10110] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:08:36.847 INFO 26087 --- [ocp-schedules-13,27363fc59c874e08,d9ad0a1565e10110] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:08:36.847 INFO 26087 --- [ocp-schedules-13,27363fc59c874e08,d9ad0a1565e10110] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:08:37.088 INFO 26087 --- [ocp-schedules-24,de852d75ca2bf721,34b8e64347177242] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:08:37.089 INFO 26087 --- [ocp-schedules-24,de852d75ca2bf721,34b8e64347177242] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:08:37.178 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,023863004b6607ba] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000016, original=READY 2025-04-08 21:08:37.180 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,023863004b6607ba] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000016, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:37.191 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,b217e750c9942218] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:37.202 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,b217e750c9942218] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000016, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:37.178+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000016}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:37.208 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,b217e750c9942218] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000016, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:37.209 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,b217e750c9942218] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000029] 2025-04-08 21:08:37.433 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,9392fab736abd680,4077c7f90bab88dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=9392fab736abd680, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:37.443 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,9392fab736abd680,4077c7f90bab88dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=9392fab736abd680, duration=14 ms] 2025-04-08 21:08:37.938 INFO 26087 --- [ocp-schedules-17,9822894f62b58c66,f53929e0e080bf70] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:08:38.186 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,8bbe67a7bf9713f9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000029, original=READY 2025-04-08 21:08:38.187 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,8bbe67a7bf9713f9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000029, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:38.194 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:38.195 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:38.206 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,be25f4f73592da08] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:38.211 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,be25f4f73592da08] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000029, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:38.185+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000029}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:38.223 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,be25f4f73592da08] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000029, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:38.224 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,be25f4f73592da08] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000030] 2025-04-08 21:08:38.568 WARN 26087 --- [ocp-async-7,3293b232287d37e4,11d783ee03d249f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:38.568 INFO 26087 --- [ocp-async-8,70054cda9d140445,2ce8156a47d1f5b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:38.647 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,22f24f5bb204e222,5b2d9a7891490b86] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=22f24f5bb204e222, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:38.657 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,22f24f5bb204e222,5b2d9a7891490b86] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=22f24f5bb204e222, duration=13 ms] 2025-04-08 21:08:39.196 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4b233edf900a0fe6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000030, original=READY 2025-04-08 21:08:39.197 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,4b233edf900a0fe6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000030, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:39.208 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,fbc9d4cd86cb860b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:39.211 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,fbc9d4cd86cb860b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000030, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:39.196+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000030}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:39.214 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,fbc9d4cd86cb860b] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 21:08:39.228 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,fbc9d4cd86cb860b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000030, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:39.229 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,fbc9d4cd86cb860b] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000026] 2025-04-08 21:08:39.355 INFO 26087 --- [prometheus-minute-schedule-1,976be7828904333e,6eae006b3faf121c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:39.846 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2da54cd67eca5915,2130a8a1a91b0907] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=2da54cd67eca5915, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:39.856 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2da54cd67eca5915,2130a8a1a91b0907] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=2da54cd67eca5915, duration=14 ms] 2025-04-08 21:08:40.001 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:08:40.001 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:08:40.205 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,fc8a5919117cd3d7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000026, original=READY 2025-04-08 21:08:40.207 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,fc8a5919117cd3d7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000026, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:40.260 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,707b67c98dde23cd] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:40.264 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,707b67c98dde23cd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000026, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:40.205+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000026}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:40.290 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,707b67c98dde23cd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000026, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:40.291 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,707b67c98dde23cd] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000031] 2025-04-08 21:08:40.568 WARN 26087 --- [ocp-async-6,47dd581dbf30e1ba,40e70b476b40e74e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:40.568 INFO 26087 --- [ocp-async-2,b72b83146569051b,93fdd5f097678c9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:41.054 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,fb77e3d878a3b9ab,b12a5b78051aad0b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=fb77e3d878a3b9ab, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:41.063 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,fb77e3d878a3b9ab,b12a5b78051aad0b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=fb77e3d878a3b9ab, duration=13 ms] 2025-04-08 21:08:41.195 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:41.195 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:41.255 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b3db743bc95d605d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000031, original=READY 2025-04-08 21:08:41.256 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b3db743bc95d605d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000031, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:41.266 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6cce0dcea217c8d0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:41.279 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6cce0dcea217c8d0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000031, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:41.254+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000031}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:41.281 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6cce0dcea217c8d0] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 21:08:41.290 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6cce0dcea217c8d0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000031, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:41.291 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,6cce0dcea217c8d0] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000033] 2025-04-08 21:08:42.261 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,d6ed0c881cb5c2a2,84c241e02ab92b8b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=d6ed0c881cb5c2a2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:42.264 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,031d7aec907fa0b3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000033, original=READY 2025-04-08 21:08:42.266 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,031d7aec907fa0b3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000033, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:42.271 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,d6ed0c881cb5c2a2,84c241e02ab92b8b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=d6ed0c881cb5c2a2, duration=14 ms] 2025-04-08 21:08:42.287 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,26d28fd27fa62f04] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:42.291 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,26d28fd27fa62f04] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000033, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:42.264+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000033}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:42.307 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,26d28fd27fa62f04] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000033, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:42.309 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,26d28fd27fa62f04] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000041] 2025-04-08 21:08:42.355 INFO 26087 --- [prometheus-minute-schedule-1,8830371f23aaf1e0,0cbe61c427f701eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:42.568 WARN 26087 --- [ocp-async-3,a77726df14794b71,cf51011086760bee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:42.568 INFO 26087 --- [ocp-async-4,a512660f31bfeec6,c7da15d3bc688930] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:43.283 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,1aad04782546b79f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000041, original=READY 2025-04-08 21:08:43.284 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,1aad04782546b79f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000041, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:43.295 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,e2453dfd095328b2] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:43.307 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,e2453dfd095328b2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000041, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:43.283+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000041}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:43.346 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,e2453dfd095328b2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000041, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:43.348 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,e2453dfd095328b2] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000043] 2025-04-08 21:08:43.460 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0713f099dc565e62,aaee347df77008b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=0713f099dc565e62, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:43.470 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0713f099dc565e62,aaee347df77008b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=0713f099dc565e62, duration=15 ms] 2025-04-08 21:08:44.195 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:44.196 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:44.290 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,c5773c18dab0aa64] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000043, original=READY 2025-04-08 21:08:44.291 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,c5773c18dab0aa64] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000043, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:44.311 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,04f6c5f472943d53] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:44.316 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,04f6c5f472943d53] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000043, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:44.290+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000043}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:44.354 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,04f6c5f472943d53] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000043, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:44.355 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,04f6c5f472943d53] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000040] 2025-04-08 21:08:44.569 WARN 26087 --- [ocp-async-10,86ec2e897a42acbf,32c4e4ba03741328] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:44.569 INFO 26087 --- [ocp-async-7,063e69bcc40339c0,bb55e46461fb749a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:44.658 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c718763ba032332,d4e35884a2d271db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=8c718763ba032332, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:44.668 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c718763ba032332,d4e35884a2d271db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=8c718763ba032332, duration=13 ms] 2025-04-08 21:08:45.299 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,5a6921f9d2cf815c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000040, original=READY 2025-04-08 21:08:45.301 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,5a6921f9d2cf815c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000040, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:45.309 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:45.313 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000040, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:45.299+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000040}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:45.315 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=1000001 2025-04-08 21:08:45.355 INFO 26087 --- [prometheus-minute-schedule-1,c2b986e30b96f79d,f6c699a450528da4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:45.353 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000001, method=deleteHost 2025-04-08 21:08:45.376 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 1000001, please retry manually 2025-04-08 21:08:45.385 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=1000001 2025-04-08 21:08:45.388 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2025-04-08 21:08:45.396 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000001 2025-04-08 21:08:45.409 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 1000001 2025-04-08 21:08:45.415 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [1000001] 2025-04-08 21:08:45.420 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [1000001] 2025-04-08 21:08:45.438 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [1000001] 2025-04-08 21:08:45.449 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [1000001] 2025-04-08 21:08:45.461 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [1000001:192.168.1.61], from status [NEW] to [NEW] 2025-04-08 21:08:45.484 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000040, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:45.486 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,4c83d2b10ffd91e0] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000034] 2025-04-08 21:08:45.858 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,22c964fee6aeb542,8efa5b19f58b6ae4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=22c964fee6aeb542, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:45.868 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,22c964fee6aeb542,8efa5b19f58b6ae4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=22c964fee6aeb542, duration=13 ms] 2025-04-08 21:08:46.306 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,1aefb2f91a4d0d99] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000034, original=READY 2025-04-08 21:08:46.308 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,1aefb2f91a4d0d99] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000034, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:08:46.323 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:08:46.342 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000034, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000006, ob_connect_address=192.168.1.61:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1000001, ob_cluster_id=1744114127, cluster_id=1000001, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:08:46.306+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=1000034}, listMap={add_region_ids=[1000002], server_ids=[1000001], add_idc_ids=[1000001], all_host_ids=[1000001], add_host_ids=[1000001], host_ids=[1000001], zone_names=[zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:08:46.350 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster metadata 2025-04-08 21:08:46.370 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.o.i.c.l.CredentialServiceListener : Detect cluster been deleted, begin to delete credentials of this cluster, clusterId=1000001 2025-04-08 21:08:46.425 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.c.CredentialServiceImpl : Deleting credential: 1000003 2025-04-08 21:08:46.454 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.c.CredentialServiceImpl : Delete credential 1000003 success 2025-04-08 21:08:46.465 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.o.i.c.l.CredentialServiceListener : Delete credentials success, clusterId=1000001 2025-04-08 21:08:46.467 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.o.i.c.l.ClusterServiceListener : detect cluster been deleted, begin to delete parameter change and compaction info, clusterId=1000001 2025-04-08 21:08:46.521 INFO 26087 --- [ocp-schedules-25,5ef88f58c1ff0107,90527e2f5fe4cc0b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:08:46.521 INFO 26087 --- [ocp-schedules-9,fc90f02de84d3400,c3185d5ca9f6c64d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:08:46.521 INFO 26087 --- [ocp-schedules-9,fc90f02de84d3400,c3185d5ca9f6c64d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:08:46.523 INFO 26087 --- [ocp-schedules-4,db04089193e6e012,29f5f677424c7cbb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:08:46.523 INFO 26087 --- [ocp-schedules-4,db04089193e6e012,29f5f677424c7cbb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:08:46.523 INFO 26087 --- [ocp-schedules-28,4e9a7224e7892c1e,821c0f80a317ae09] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:08:46.524 INFO 26087 --- [ocp-schedules-28,4e9a7224e7892c1e,821c0f80a317ae09] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:08:46.524 INFO 26087 --- [ocp-schedules-28,a2e0ab91cf556f4d,6248ac37243e76bb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:08:46.524 INFO 26087 --- [ocp-schedules-13,a69811ca76e69b25,cb00f518ac278aca] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:08:46.525 INFO 26087 --- [ocp-schedules-28,a2e0ab91cf556f4d,6248ac37243e76bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:46.525 INFO 26087 --- [ocp-schedules-13,a69811ca76e69b25,cb00f518ac278aca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:08:46.542 INFO 26087 --- [ocp-async-8,0079591b8adea732,20cc4541de0270a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:08:46.543 INFO 26087 --- [ocp-schedules-16,88f3726b6e1ceb2a,6a8f013e0a9d7c9b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:08:46.543 INFO 26087 --- [ocp-schedules-16,7ff5c132b1afa841,bbf651670eb54aa1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:08:46.544 INFO 26087 --- [ocp-schedules-16,7ff5c132b1afa841,bbf651670eb54aa1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:46.514 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000005, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:08:46.547 INFO 26087 --- [ocp-schedules-10,f7eccd636830dacd,1793e7e41c4b4001] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:08:46.549 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000001, status=TAKINGOVER 2025-04-08 21:08:46.549 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000001, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:07:05+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:08:46.546 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.o.i.c.l.ClusterServiceListener : detect parameter change and compaction info finished, clusterId=1000001 2025-04-08 21:08:46.558 INFO 26087 --- [ocp-schedules-2,2387da48a9afb5b7,a1a6af228691e7ca] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:08:46.558 INFO 26087 --- [ocp-schedules-2,2387da48a9afb5b7,a1a6af228691e7ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000001] 2025-04-08 21:08:46.559 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:08:46.561 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000006, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["ocpdb"]} 2025-04-08 21:08:46.569 WARN 26087 --- [ocp-async-6,abbef95ac1f83c55,3e3f873fe5fbfc24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:46.569 INFO 26087 --- [ocp-async-1,7718fe0f2e3407d3,7d8db7ecb4735f2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:46.567 ERROR 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000001] 2025-04-08 21:08:46.583 INFO 26087 --- [ocp-schedules-11,82daf13b9285de67,60b22f5a66a86e5d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:08:46.593 INFO 26087 --- [ocp-schedules-22,25e52dc8e2408924,5880719e6a9b74ed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:08:46.572 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000001 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$5d1537be.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:08:46.617 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 21:08:46.617 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:08:46.617 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 21:08:46.619 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db success. 2025-04-08 21:08:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:08:46.623245+08:00, lastProcessTime=2025-04-08T21:08:36.623537+08:00 2025-04-08 21:08:46.607 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.o.internal.ObproxyServiceListener : detect cluster been deleted, begin to delete oblinks, obclusterName=ocpdb, obClusterId=1744114127 2025-04-08 21:08:46.652 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000005, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:08:46.652 INFO 26087 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Receive invalidate ob version event, clusters:[ocpdb] 2025-04-08 21:08:46.653 INFO 26087 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Invalidate ob version success, clusters:[ocpdb] 2025-04-08 21:08:46.664 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000006, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["ocpdb"]} 2025-04-08 21:08:46.665 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000007, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:08:46.667 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.b.i.l.BackupServiceListener : detect cluster been deleted, begin to delete backup strategy, clusterId=1000001 2025-04-08 21:08:46.688 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.b.i.l.BackupServiceListener : detect backup strategy finished, clusterId=1000001 2025-04-08 21:08:46.704 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.s.o.OrganizationResourceManager : delete resource for current organization,organizationId=10000000,resourceType=1000001, resourceId=1000001 2025-04-08 21:08:46.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:08:46.623245+08:00 2025-04-08 21:08:46.715 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster finished 2025-04-08 21:08:46.723 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host started, hostIds=[1000001] 2025-04-08 21:08:46.732 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000001, method=deleteHost 2025-04-08 21:08:46.747 ERROR 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000001] 2025-04-08 21:08:46.754 WARN 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 1000001 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000001 2025-04-08 21:08:46.761 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host finished 2025-04-08 21:08:46.767 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.host.service.IdcServiceImpl : Deleted idc: id=1000001 2025-04-08 21:08:46.772 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.host.service.RegionServiceImpl : Deleted region: 1000002 deault_region 2025-04-08 21:08:46.788 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : rollback finished 2025-04-08 21:08:46.795 INFO 26087 --- [manual-subtask-executor16,34015dfa7d12dcff,94cdb8689028f67e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000034, operation:ROLLBACK, state: PENDING 2025-04-08 21:08:47.076 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,0ff7ffc9200f24d5,f67e012e1d287b21] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=0ff7ffc9200f24d5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:47.086 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,0ff7ffc9200f24d5,f67e012e1d287b21] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=0ff7ffc9200f24d5, duration=14 ms] 2025-04-08 21:08:47.091 INFO 26087 --- [ocp-schedules-20,d72103b4a4b6a5e7,7f4049a90a3badd0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:08:47.092 INFO 26087 --- [ocp-schedules-20,d72103b4a4b6a5e7,7f4049a90a3badd0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:08:47.196 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:47.197 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:47.364 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000006 2025-04-08 21:08:47.364 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000005 2025-04-08 21:08:47.502 ERROR 26087 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : GetClusterOverview, clusterId:1000001, NotFoundException {} com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1000001 at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl.nullSafeGetClusterOverview(ClusterDaoManagerImpl.java:94) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$FastClassBySpringCGLIB$$1f10cbd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$EnhancerBySpringCGLIB$$5edf77dc.nullSafeGetClusterOverview() at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:126) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:08:47.502 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:08:47.665 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 21:08:47.665 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:08:47.665 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 21:08:47.665 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster ocpdb not found in config server 2025-04-08 21:08:47.682 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000007, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:08:47.980 INFO 26087 --- [ocp-schedules-11,99e9bb60c7ccd304,a11fdba56fc261aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:08:48.355 INFO 26087 --- [prometheus-minute-schedule-1,aa86698b2cb13212,5d18c1089647638e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:48.355 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,88a96a7f589af192,3fe8fc955d98e059] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000006?taskInstanceId=1000006, client=10.11.12.102, traceId=88a96a7f589af192, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000006,] 2025-04-08 21:08:48.364 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,88a96a7f589af192,3fe8fc955d98e059] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000006 client=10.11.12.102, traceId=88a96a7f589af192, duration=13 ms] 2025-04-08 21:08:48.365 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000007 2025-04-08 21:08:48.569 WARN 26087 --- [ocp-async-3,b6af1380ba85dbe4,12bd6aaa9a9783d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:48.569 INFO 26087 --- [ocp-async-5,558ed819a5dd7afa,01812cae7145de90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:48.885 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,187d169eaed422dd,13340d079d9b276d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=187d169eaed422dd, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:08:48.885 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,4ddea910072c6e6a,9378841339ed190a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=4ddea910072c6e6a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:08:48.885 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,5fb1aa3d309e3579,864db834912c9321] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=5fb1aa3d309e3579, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:08:48.885 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,4ddea910072c6e6a,9378841339ed190a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=4ddea910072c6e6a, duration=3 ms] 2025-04-08 21:08:48.888 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,e17c7dca472de4b3,ee46cbe36b11c27d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=e17c7dca472de4b3, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:08:48.888 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,9139c96c170e7e74,d7023ccee79321e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A08%3A48%2B08%3A00&endTime=2025-04-08T21%3A08%3A48%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=9139c96c170e7e74, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:08:48+08:00,2025-04-08T21:08:48+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:08:48.888 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,187d169eaed422dd,13340d079d9b276d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=187d169eaed422dd, duration=6 ms] 2025-04-08 21:08:48.888 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,e17c7dca472de4b3,ee46cbe36b11c27d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=e17c7dca472de4b3, duration=5 ms] 2025-04-08 21:08:48.891 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,5fb1aa3d309e3579,864db834912c9321] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=5fb1aa3d309e3579, duration=9 ms] 2025-04-08 21:08:48.893 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,9139c96c170e7e74,d7023ccee79321e4] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:08:48.893 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,9139c96c170e7e74,d7023ccee79321e4] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114140 2025-04-08 21:08:48.894 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,9139c96c170e7e74,d7023ccee79321e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=9139c96c170e7e74, duration=12 ms] 2025-04-08 21:08:48.908 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,e7ead10991ea887e,92e9b2a096d1ee08] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=e7ead10991ea887e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:08:48.914 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,e7ead10991ea887e,92e9b2a096d1ee08] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=e7ead10991ea887e, duration=9 ms] 2025-04-08 21:08:49.862 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,93abae9f08a59532,046227b817bfff1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=93abae9f08a59532, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:08:49.865 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,93abae9f08a59532,046227b817bfff1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=93abae9f08a59532, duration=6 ms] 2025-04-08 21:08:50.197 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:50.198 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:50.569 WARN 26087 --- [ocp-async-10,340db36eb7b5f7f0,dcd8eadc949a7bd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:50.569 INFO 26087 --- [ocp-async-9,f0b5aacc2eb74eb9,1a6dee820262fc2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:50.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,9570ea994f2669cb,be81528ede990c1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=9570ea994f2669cb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:08:50.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,5a764da76b0c5c48,994b19dc84942ff2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=5a764da76b0c5c48, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:08:50.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,5a764da76b0c5c48,994b19dc84942ff2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=5a764da76b0c5c48, duration=2 ms] 2025-04-08 21:08:50.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,08f707f1a8d37db1,7d8dd55067bd82bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=08f707f1a8d37db1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:08:50.957 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,08f707f1a8d37db1,7d8dd55067bd82bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=08f707f1a8d37db1, duration=3 ms] 2025-04-08 21:08:50.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,427ede4b7ae1c511,f750914dd3b74060] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=427ede4b7ae1c511, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:08:50.959 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,9570ea994f2669cb,be81528ede990c1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=9570ea994f2669cb, duration=5 ms] 2025-04-08 21:08:51.022 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,e971da081aff3912,171cd0d908290f79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=e971da081aff3912, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:08:51.025 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,e971da081aff3912,171cd0d908290f79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=e971da081aff3912, duration=7 ms] 2025-04-08 21:08:51.042 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,427ede4b7ae1c511,f750914dd3b74060] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=427ede4b7ae1c511, duration=88 ms] 2025-04-08 21:08:51.100 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:08:51.100 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:08:51.100 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:08:51.355 INFO 26087 --- [prometheus-minute-schedule-1,43cedaada6dc9b5d,6d0908aae9d79d71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:52.569 WARN 26087 --- [ocp-async-6,2b1b9b593a608c2e,f4e0578d9196dd90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:52.569 INFO 26087 --- [ocp-async-8,01ecddcbd23aec74,4fc2778eb944ffff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:52.951 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,ddd0bffb0feea55a,80a61fc25ed846f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=ddd0bffb0feea55a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:08:52.951 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b3ed55c81a5ffa8f,99ff38748037bc4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=b3ed55c81a5ffa8f, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:08:52.951 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,b3ed55c81a5ffa8f,99ff38748037bc4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=b3ed55c81a5ffa8f, duration=3 ms] 2025-04-08 21:08:52.951 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,ddd0bffb0feea55a,80a61fc25ed846f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=ddd0bffb0feea55a, duration=3 ms] 2025-04-08 21:08:52.952 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,003e59f45df646da,6eef2fed977eb340] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=003e59f45df646da, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:08:52.952 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,003e59f45df646da,6eef2fed977eb340] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=003e59f45df646da, duration=4 ms] 2025-04-08 21:08:52.973 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:08:52.973 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:08:52.973 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:08:53.198 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:53.198 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:54.355 INFO 26087 --- [prometheus-minute-schedule-1,1724b9955e15e1a8,dfc0ddeee5f8dfca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:54.569 WARN 26087 --- [ocp-async-4,0ebc04283e4f831c,714c9d6112dca7f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:54.569 INFO 26087 --- [ocp-async-8,94b1fe2c84a916b1,90ad511b17c7ffd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:56.198 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:56.199 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:08:56.526 INFO 26087 --- [ocp-schedules-13,a69811ca76e69b25,cb00f518ac278aca] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:08:56.526 INFO 26087 --- [ocp-schedules-28,a2e0ab91cf556f4d,6248ac37243e76bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:08:56.542 INFO 26087 --- [ocp-async-8,51b6dcf4288f4c69,5ea228da9cd48f1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:08:56.559 INFO 26087 --- [ocp-schedules-2,2387da48a9afb5b7,a1a6af228691e7ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:08:56.570 WARN 26087 --- [ocp-async-10,4f8d7d0ace835670,d0a3463eabb50d96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:56.570 INFO 26087 --- [ocp-async-9,7cb2ccccaed7297f,36d07bc3b503314d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:56.594 INFO 26087 --- [ocp-schedules-5,08629a4e4c2d5d1b,e70e0c48e5bac5b3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:08:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:08:56.623326+08:00, lastProcessTime=2025-04-08T21:08:46.623245+08:00 2025-04-08 21:08:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb 2025-04-08 21:08:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:08:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb 2025-04-08 21:08:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:08:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:08:56.740 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:08:56.623326+08:00 2025-04-08 21:08:56.792 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,e9788cf21911ebd4,8f968517f75cc66e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=e9788cf21911ebd4, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:08:56.794 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,e9788cf21911ebd4,8f968517f75cc66e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=e9788cf21911ebd4, duration=5 ms] 2025-04-08 21:08:56.848 INFO 26087 --- [ocp-schedules-26,e4b41b65cd515331,bc58ef10da106ced] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:08:56.851 INFO 26087 --- [ocp-schedules-26,e4b41b65cd515331,bc58ef10da106ced] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:08:56.851 INFO 26087 --- [ocp-schedules-26,e4b41b65cd515331,bc58ef10da106ced] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:08:57.097 INFO 26087 --- [ocp-schedules-15,974adad20de4bc77,05e8ca77ddb4215a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:08:57.098 INFO 26087 --- [ocp-schedules-15,974adad20de4bc77,05e8ca77ddb4215a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:08:57.355 INFO 26087 --- [prometheus-minute-schedule-1,f70aa0ed85db1550,2d8337f302f0da92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117740, interval=60 2025-04-08 21:08:58.001 INFO 26087 --- [ocp-schedules-1,bf853bc997071ad7,f40110d612f1ba13] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:08:58.570 WARN 26087 --- [ocp-async-1,d0093a0e4b52312c,6d7008d8381c9f43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:08:58.570 INFO 26087 --- [ocp-async-6,0b3edf733bcd6de8,13ee2d3ce9887634] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:08:59.199 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:08:59.200 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:00.355 INFO 26087 --- [prometheus-minute-schedule-1,0c09b12031d3004a,0a66ea029e457056] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:00.570 WARN 26087 --- [ocp-async-4,022dba2e8b9e2aae,30bd81f3b1dff786] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:00.570 INFO 26087 --- [ocp-async-5,a008d9edb8b9035e,dc9aae635024f67d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:02.200 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:02.201 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:02.570 WARN 26087 --- [ocp-async-7,83b6bc6d36613093,046b7b001a543c4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:02.570 INFO 26087 --- [ocp-async-8,66ac48e6a2d36299,808551ba65899f40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:03.355 INFO 26087 --- [prometheus-minute-schedule-1,1a678027bfb3e435,a025032166959f6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:04.570 WARN 26087 --- [ocp-async-1,b3e881aa5e5bf7bc,60c1dfd3ce10150d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:04.570 INFO 26087 --- [ocp-async-2,89b741330ca28a8f,6e7982abf076e405] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:05.201 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:05.202 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:06.205 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:03:06.167, end=2025-04-08T21:08:06.202} 2025-04-08 21:09:06.207 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:09:06.208 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:09:06.209 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:03:06.157, end=2025-04-08T21:08:06.206} 2025-04-08 21:09:06.211 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:09:06.211 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:09:06.355 INFO 26087 --- [prometheus-minute-schedule-1,bd0717c708dcf1ec,2098b776ae80c372] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:06.355 INFO 26087 --- [prometheus-cache-monitor1,320027a2f8e50213,ac982d0a5e5bd19d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1104,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":372} 2025-04-08 21:09:06.355 INFO 26087 --- [prometheus-cache-monitor1,320027a2f8e50213,ac982d0a5e5bd19d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:09:06.543 INFO 26087 --- [ocp-async-2,ca83cc7726d72ecb,8cd5714734696b33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:09:06.570 WARN 26087 --- [ocp-async-10,1eb602cf6d373247,4f951fb539873d4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:06.570 INFO 26087 --- [ocp-async-3,263a80a6aa1650a5,83b4df0a810d1394] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:06.596 INFO 26087 --- [ocp-schedules-12,b092b23820cbe17f,0928790396b951ae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:09:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:09:06.623681+08:00, lastProcessTime=2025-04-08T21:08:56.623326+08:00 2025-04-08 21:09:06.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:09:06.623681+08:00 2025-04-08 21:09:07.102 INFO 26087 --- [ocp-schedules-28,60d4b5f2705d8ce1,bd0c15979904a166] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:09:07.103 INFO 26087 --- [ocp-schedules-28,60d4b5f2705d8ce1,bd0c15979904a166] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:09:08.035 INFO 26087 --- [ocp-schedules-4,3e69b48cb1a95d42,854c51bb5a08baaf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:09:08.202 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:08.202 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:08.570 WARN 26087 --- [ocp-async-8,dc386723e1084861,e0e7566270a7a0c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:08.571 INFO 26087 --- [ocp-async-5,ab6ec536e1e7c7b5,573d293d6045c35a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:09.355 INFO 26087 --- [prometheus-minute-schedule-1,5873b78508dd7d67,be89a71f41df0e36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:10.001 INFO 26087 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:09:10.001 INFO 26087 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:09:10.571 WARN 26087 --- [ocp-async-1,af423fc76940ab18,d35522afb15503e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:10.571 INFO 26087 --- [ocp-async-9,fd44676264fe521c,31cf817d2f9a4ea7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:09:11.203 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:11.203 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:12.355 INFO 26087 --- [prometheus-minute-schedule-1,be299aabd042847e,153a48fa99caf32c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:12.361 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117680, nodeCount=374, affectRows=374 2025-04-08 21:09:12.363 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117680, nodeCount=371, validCount=371 2025-04-08 21:09:12.571 WARN 26087 --- [ocp-async-9,1606699febca7939,4dcaba155f856208] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:12.571 INFO 26087 --- [ocp-async-10,1d83a60e0d7b9717,0108aa81953c992a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:14.203 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:14.204 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:14.571 WARN 26087 --- [ocp-async-10,eb4b9223144c3aae,6d9eff37be717dbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:14.571 INFO 26087 --- [ocp-async-8,2db0a3edc83e28d2,db0ef277aa659bb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:15.355 INFO 26087 --- [prometheus-minute-schedule-1,e7c457f5ac200583,663ba987fbf99311] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:16.517 INFO 26087 --- [ocp-schedules-12,ae560cfa5bd15828,5f316a4f7696f0e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:09:16.518 INFO 26087 --- [ocp-schedules-1,2f492a13ab646b45,4c6d9339d0d0df38] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:09:16.521 INFO 26087 --- [ocp-schedules-11,76ed0f5214014af2,6ccb8a6345a178dd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:09:16.521 INFO 26087 --- [ocp-schedules-27,f4d3e7381a12abd5,203b397948585cc1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:09:16.521 INFO 26087 --- [ocp-schedules-27,f4d3e7381a12abd5,203b397948585cc1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:09:16.523 INFO 26087 --- [ocp-schedules-25,a43aedf95cf06f00,8e2124f3a6bf13c3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:09:16.523 INFO 26087 --- [ocp-schedules-29,57d24b300dfa80cb,4b8940bb2aef7a39] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:09:16.523 INFO 26087 --- [ocp-schedules-25,a43aedf95cf06f00,8e2124f3a6bf13c3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:09:16.523 INFO 26087 --- [ocp-schedules-29,57d24b300dfa80cb,4b8940bb2aef7a39] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:09:16.524 INFO 26087 --- [ocp-schedules-14,9a37b6001c2c52e5,58c3f7c4b16a8ef4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:09:16.524 INFO 26087 --- [ocp-schedules-15,7d618619f72c6994,02e7143ac7cf4deb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:09:16.526 INFO 26087 --- [ocp-schedules-1,2f492a13ab646b45,4c6d9339d0d0df38] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:09:16.530 INFO 26087 --- [ocp-schedules-12,ae560cfa5bd15828,5f316a4f7696f0e6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:09:16.532 INFO 26087 --- [ocp-schedules-24,bbced5db79698993,63f2e57a615194b0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:09:16.543 INFO 26087 --- [ocp-async-1,3bbc968009885331,574a9d687c400a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:09:16.549 INFO 26087 --- [ocp-schedules-16,7ff5c132b1afa841,bbf651670eb54aa1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:09:16.549 INFO 26087 --- [ocp-schedules-16,d16d2b24f6d8092a,5af7feeb24b107a9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:09:16.571 WARN 26087 --- [ocp-async-4,c4d1f9f7b7341b49,6f82e079bdc82e56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:16.571 INFO 26087 --- [ocp-async-9,e4b92c96fd28317f,4586cc315854a384] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:16.578 INFO 26087 --- [ocp-schedules-7,53544e1d2dd70306,f5dde8638e3b6f76] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:09:16.584 INFO 26087 --- [ocp-schedules-8,46ed1c09cb96c2bd,5d216464eee5cd25] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1478,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":372,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":372} 2025-04-08 21:09:16.584 INFO 26087 --- [ocp-schedules-8,46ed1c09cb96c2bd,5d216464eee5cd25] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:09:16.585 INFO 26087 --- [ocp-schedules-27,fa24211e14f118d0,203337960c9b751b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:09:16.589 INFO 26087 --- [ocp-schedules-25,872a476dd42703e9,b0fbc71aaff0d56f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:09:16.589+08:00, clusters=[] 2025-04-08 21:09:16.590 INFO 26087 --- [ocp-schedules-29,468338aeb4d0b77b,f7dbe5872a544c0f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:09:16.590 INFO 26087 --- [ocp-schedules-15,3821b48c299f23e8,ce5a7146da28d146] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:09:16.590+08:00, clusters=[] 2025-04-08 21:09:16.597 INFO 26087 --- [ocp-schedules-14,44ee76f0829b7555,7676a8334b31a52a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:09:16.600 INFO 26087 --- [ocp-schedules-1,a3e952ad6d384174,8633e89940fa4deb] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:09:16.600 INFO 26087 --- [ocp-schedules-1,a3e952ad6d384174,8633e89940fa4deb] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:09:16.618 INFO 26087 --- [ocp-schedules-24,f9cd8a21df0ada1a,d2f470dddbb95e3d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:09:16.618 INFO 26087 --- [ocp-schedules-24,f9cd8a21df0ada1a,d2f470dddbb95e3d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:09:16.619 INFO 26087 --- [ocp-schedules-24,f9cd8a21df0ada1a,d2f470dddbb95e3d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:09:16.619 INFO 26087 --- [ocp-schedules-24,f9cd8a21df0ada1a,d2f470dddbb95e3d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:09:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:09:16.622596+08:00, lastProcessTime=2025-04-08T21:09:06.623681+08:00 2025-04-08 21:09:16.624 INFO 26087 --- [ocp-schedules-2,93a51fe2981e15bf,dc3d15f16fce46ec] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:09:16.624 INFO 26087 --- [ocp-schedules-2,93a51fe2981e15bf,dc3d15f16fce46ec] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:09:16.630 INFO 26087 --- [ocp-schedules-17,d95c42a140c8c317,45d5ae1803fc2dc5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:09:16.650 INFO 26087 --- [ocp-schedules-20,d2f5c4949f5d19c3,d5ace6e4c04caa9b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:09:16.655 INFO 26087 --- [ocp-schedules-28,63adec1a2e39a463,fc8c86d805a1d844] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:09:16.656 INFO 26087 --- [ocp-schedules-28,63adec1a2e39a463,fc8c86d805a1d844] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:09:16.656 INFO 26087 --- [ocp-schedules-28,63adec1a2e39a463,fc8c86d805a1d844] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:09:16.668 INFO 26087 --- [ocp-schedules-16,6c2795d9f023f853,cfe1e2ad65eb8471] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:09:16.729 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:09:16.622596+08:00 2025-04-08 21:09:16.780 INFO 26087 --- [ocp-schedules-6,72f9db628a5e092d,7cb8d9c71d976f4e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:09:16.804 INFO 26087 --- [ocp-schedules-6,20cab46a4a452427,6120aa9a95b3ff2c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:09:16.804 INFO 26087 --- [ocp-schedules-6,20cab46a4a452427,6120aa9a95b3ff2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:09:16.804 INFO 26087 --- [ocp-schedules-6,20cab46a4a452427,6120aa9a95b3ff2c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:09:16.804 INFO 26087 --- [ocp-schedules-6,20cab46a4a452427,6120aa9a95b3ff2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:09:16.804 INFO 26087 --- [ocp-schedules-6,20cab46a4a452427,6120aa9a95b3ff2c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:09:16.804 INFO 26087 --- [ocp-schedules-6,20cab46a4a452427,6120aa9a95b3ff2c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:09:16.854 INFO 26087 --- [ocp-schedules-30,cb364a71775bd48b,fa21bf55da4ef71c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:09:16.878 INFO 26087 --- [ocp-schedules-30,cb364a71775bd48b,fa21bf55da4ef71c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:09:16.878 INFO 26087 --- [ocp-schedules-30,cb364a71775bd48b,fa21bf55da4ef71c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:09:16.972 INFO 26087 --- [ocp-schedules-10,e21bfc9f966e9a5b,077c0f902d30a972] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61, current targets ,,,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61 2025-04-08 21:09:17.107 INFO 26087 --- [ocp-schedules-22,f0f55f85e352db28,16ca843dbd83662f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:09:17.108 INFO 26087 --- [ocp-schedules-22,f0f55f85e352db28,16ca843dbd83662f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:09:17.127 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:09:17.128 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:09:17.129 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:09:17.130 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:09:17.131 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:09:17.132 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:09:17.133 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:09:17.134 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:09:17.135 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:09:17.136 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:09:17.137 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:09:17.138 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:09:17.139 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:09:17.140 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:09:17.141 INFO 26087 --- [ocp-schedules-26,b70c54ff06a46723,a30b43bef46e82a6] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:09:17.204 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:17.205 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:17.367 INFO 26087 --- [ocp-schedules-8,f22a6f8910863859,f9de312e80642e14] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:09:17.367 INFO 26087 --- [ocp-schedules-8,f22a6f8910863859,f9de312e80642e14] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:09:18.056 INFO 26087 --- [ocp-schedules-15,0323716952d15d04,ce0b5f33dc5220d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:09:18.355 INFO 26087 --- [prometheus-minute-schedule-1,e35a0e0ead4fe70f,07fa443e5bf4a656] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:18.571 WARN 26087 --- [ocp-async-3,b76a64df6301a9b1,1954ed146a7538d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:18.571 INFO 26087 --- [ocp-async-9,c805acaefc47a88b,838b87e2bd879671] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:19.278 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:09:19.278 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:09:19.281 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:09:19.281 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:09:19.281 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:09:19.281 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:09:19.285 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:09:19.285 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:09:19.285 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:09:19.285 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:09:19.310 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:09:19.316 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:09:19.316 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:09:19.316 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:09:19.316 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:09:19.316 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:09:19.322 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:09:19.637 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:09:19.968 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.682991875S 2025-04-08 21:09:19.968 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:09:20.024 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:09:20.024 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:09:20.048 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:09:20.096 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:09:20.096 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:09:20.096 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:09:20.096 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:09:20.096 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:09:20.097 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:09:20.104 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:09:20.205 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:20.206 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:20.225 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] o.s.boot.SpringApplication : Started application in 0.944 seconds (JVM running for 465.598) 2025-04-08 21:09:20.225 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.944253014S 2025-04-08 21:09:20.498 INFO 26087 --- [ocp-schedules-24,31dfd9add18019d5,87439fd782792e57] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:09:20.571 INFO 26087 --- [ocp-async-7,0694af5c6c8b6459,44aa79415367a160] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:20.571 WARN 26087 --- [ocp-async-5,5c1c59fb4ba8be15,4b07725d4dfd0832] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:09:21.355 INFO 26087 --- [prometheus-minute-schedule-1,066dd5a12c937a28,d39f4e66fe68a13d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:22.572 WARN 26087 --- [ocp-async-1,22d7d0c308c6d7c0,a1f40616b358ecc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:22.572 INFO 26087 --- [ocp-async-4,f3a97f971f4edc7a,4243e3c13d309340] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:23.206 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:23.206 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:24.355 INFO 26087 --- [prometheus-minute-schedule-1,8723c107db1abaa0,15b0f16256815b2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:24.572 WARN 26087 --- [ocp-async-3,e8685a488da02ba1,ae283f81a9c5e928] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:24.572 INFO 26087 --- [ocp-async-4,3e65190ecdf4dd59,5ce48a576b61858a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:26.207 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:26.207 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:26.543 INFO 26087 --- [ocp-async-5,2dbf362bad879274,6b2f243ad4f0acc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:09:26.572 WARN 26087 --- [ocp-async-8,4326e5b3439a90cf,36a0961ddd19b482] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:26.572 INFO 26087 --- [ocp-async-7,51c5e093b13a02cb,efcbf205bb87d54c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:26.598 INFO 26087 --- [ocp-schedules-7,a4aaa65860a387b8,4027c15f17df9f69] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:09:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:09:26.623949+08:00, lastProcessTime=2025-04-08T21:09:16.622596+08:00 2025-04-08 21:09:26.723 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:09:26.623949+08:00 2025-04-08 21:09:27.115 INFO 26087 --- [ocp-schedules-15,0f1cfd2e90a0bc2b,80d7a5936b779c83] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:09:27.116 INFO 26087 --- [ocp-schedules-15,0f1cfd2e90a0bc2b,80d7a5936b779c83] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:09:27.355 INFO 26087 --- [prometheus-minute-schedule-1,e74477f3d85c42a1,c787c7199d6d955a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:28.146 INFO 26087 --- [ocp-schedules-6,3d0c3956d4a9191c,9705eb79e2698288] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:09:28.572 WARN 26087 --- [ocp-async-7,c1eed5146b7db22e,a28be08b864b9eac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:28.572 INFO 26087 --- [ocp-async-3,73308ae87608c3fe,fbe994646998deea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:29.207 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:29.208 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:30.355 INFO 26087 --- [prometheus-minute-schedule-1,d288600e60479ca3,c98710cf4a960c22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:30.572 WARN 26087 --- [ocp-async-9,6d4cb4e2c9d1cadf,75dde4cac874c18e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:30.572 INFO 26087 --- [ocp-async-4,0dd254129933b1ed,6071923798c975a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:32.208 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:32.209 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:32.572 WARN 26087 --- [ocp-async-2,054c12e1657d4e24,6b06ccd1ddaf9892] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:32.572 INFO 26087 --- [ocp-async-8,c5e28c829713557b,61e099799965477e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:33.355 INFO 26087 --- [prometheus-minute-schedule-1,9531a6236ed3e01d,5a9b128975b3d7d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:34.572 WARN 26087 --- [ocp-async-5,54e77c1fddf51423,4c10f199a0404221] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:34.573 INFO 26087 --- [ocp-async-6,d3de7be3635ab73a,2bfd90df3b8de1e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:34.995 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:34.995 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:34.996 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:34.996 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:09:34.997 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:09:35.209 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:35.209 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:36.355 INFO 26087 --- [prometheus-minute-schedule-1,aa46f757749b78ee,19a8996926005ccf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:36.543 INFO 26087 --- [ocp-async-3,682472c323e2f8b7,aa11219112db4275] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:09:36.573 WARN 26087 --- [ocp-async-4,008df9463699852d,fbe9d4a112527271] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:36.573 INFO 26087 --- [ocp-async-10,cfced9cb278e8a78,7cc924e7e3d97dcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:36.600 INFO 26087 --- [ocp-schedules-10,08eec6574aaff1dd,5125a020d4710344] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:09:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:09:36.622759+08:00, lastProcessTime=2025-04-08T21:09:26.623949+08:00 2025-04-08 21:09:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:09:36.622759+08:00 2025-04-08 21:09:36.880 INFO 26087 --- [ocp-schedules-4,68af2c6e7b9a61fb,8cd5e7ca1e4c41ff] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:09:36.897 INFO 26087 --- [ocp-schedules-4,68af2c6e7b9a61fb,8cd5e7ca1e4c41ff] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:09:36.897 INFO 26087 --- [ocp-schedules-4,68af2c6e7b9a61fb,8cd5e7ca1e4c41ff] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:09:37.120 INFO 26087 --- [ocp-schedules-16,20b08b034839d9d7,dc6a3f1cea5a277b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:09:37.122 INFO 26087 --- [ocp-schedules-16,20b08b034839d9d7,dc6a3f1cea5a277b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:09:38.164 INFO 26087 --- [ocp-schedules-5,f3beb6540e84e582,5ad9030ff38d9e30] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:09:38.209 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:38.210 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:38.573 WARN 26087 --- [ocp-async-1,b66324a8dcb11745,9acd2b2ca5216150] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:38.573 INFO 26087 --- [ocp-async-2,ad383c030a7e4ff5,2ee1fddb342fb455] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:39.355 INFO 26087 --- [prometheus-minute-schedule-1,1a39f7cd9aca6c59,91a2df58b5abc8a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:40.002 INFO 26087 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:09:40.002 INFO 26087 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:09:40.573 WARN 26087 --- [ocp-async-5,36477175041f84e9,0fd2492648b9e464] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:40.573 INFO 26087 --- [ocp-async-6,dead1fab068cceff,15ea6d7e354d4ca4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:40.644 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOverPreCheck, client 10.11.12.102, traceId 298ccb9a3810258f] 2025-04-08 21:09:40.646 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOverPreCheck?null, client=10.11.12.102, traceId=298ccb9a3810258f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(TakeOverClusterPreCheckParam), args=TakeOverClusterPreCheckParam(connectionMode=direct, clusterName=null, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=192.168.1.62, port=2881),] 2025-04-08 21:09:40.649 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=null, obClusterId=null, tenantName=sys, username=root, address=192.168.1.62, port=2881, database=oceanbase) 2025-04-08 21:09:40.649 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.62, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:09:40.657 INFO 26087 --- [obsdk-init-thread-5,298ccb9a3810258f,bda8a2fb7a66afb2] com.alibaba.druid.pool.DruidDataSource : {dataSource-7,ob-connector-datasource_direct_sys_root_192.168.1.62_2881} inited 2025-04-08 21:09:40.657 INFO 26087 --- [obsdk-init-thread-5,298ccb9a3810258f,bda8a2fb7a66afb2] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.62, port=2881, obsAddrList=null, username=root, tenantName=sys, clusterName=null, obClusterId=null, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:09:40.657 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:09:40.658 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.659 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.676 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.676 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.680 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.680 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.683 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.684 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.688 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.688 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.696 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.696 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 21:09:40.721 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.721 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.725 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.726 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 21:09:40.738 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.738 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.741 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.742 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 21:09:40.753 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.754 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.757 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.757 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.765 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.765 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 21:09:40.767 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.768 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.772 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.772 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.776 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 21:09:40.790 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.790 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 21:09:40.818 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.819 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.826 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.828 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 21:09:40.834 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.835 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.840 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.841 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.850 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:09:40.850 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.856 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:09:40.907 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.908 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:40.916 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:40.917 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:09:41.003 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.004 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.008 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.008 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.012 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.013 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.016 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.017 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.020 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.021 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:09:41.072 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.073 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.081 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.081 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:09:41.094 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.095 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.099 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.100 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:09:41.158 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.158 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.162 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.163 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.167 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.168 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.176 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.176 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:09:41.210 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:41.211 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:41.228 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.229 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.233 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.234 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:09:41.241 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.242 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.246 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.247 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:09:41.304 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.305 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.309 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.310 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.313 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.313 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.317 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.317 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.321 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.322 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:09:41.374 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.375 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.379 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.381 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:09:41.393 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.393 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.397 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.398 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.402 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.403 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.406 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.407 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 21:09:41.459 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.460 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT name, value FROM `ocp`.`ops_config` WHERE name = ?, args: [ob.proxyro.password] 2025-04-08 21:09:41.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.475 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.479 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.479 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.483 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.484 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:41.485 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.485 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:41.486 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.486 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:09:41.491 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.492 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.496 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.497 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.500 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.501 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.504 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.505 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.509 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.510 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:09:41.522 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.523 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.528 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.528 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.532 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.535 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `user`, (CASE `account_locked` WHEN 'Y' THEN 1 ELSE 0 END) AS account_locked, `select_priv`, `insert_priv`, `update_priv`, `delete_priv`, `create_priv`, `drop_priv`, `process_priv`, `grant_priv`, `index_priv`, `alter_priv`, `show_db_priv`, `super_priv`, `create_view_priv`, `show_view_priv`, `create_user_priv`, `create_routine_priv`, `alter_routine_priv`, `execute_priv`, `password`, `host` FROM `mysql`.`user` WHERE `user` = binary ? AND `host` = ?, args: [proxyro, %] 2025-04-08 21:09:41.564 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.565 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW DATABASES 2025-04-08 21:09:41.568 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.569 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:41.574 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:41.575 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `db`, `user`, `select_priv`, `insert_priv`, `update_priv`, `delete_priv`, `create_priv`, `drop_priv`, `index_priv`, `alter_priv`, `create_view_priv`, `show_view_priv` , `host` FROM `mysql`.`db` WHERE `user` = binary ? AND `host` = ?, args: [proxyro, %] 2025-04-08 21:09:41.600 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.62] 2025-04-08 21:09:41.604 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.63] 2025-04-08 21:09:41.607 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.64] 2025-04-08 21:09:41.611 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.65] 2025-04-08 21:09:41.612 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,298ccb9a3810258f,20001efa2eb15fb8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.11.12.102, traceId=298ccb9a3810258f, duration=971 ms] 2025-04-08 21:09:41.729 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,03f197c4127df1d1,10eae4d301577127] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=03f197c4127df1d1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 21:09:41.729 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,8b70f0e747ab66e0,b04b87f080d6ef5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=8b70f0e747ab66e0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 21:09:41.731 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,03f197c4127df1d1,10eae4d301577127] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=03f197c4127df1d1, duration=6 ms] 2025-04-08 21:09:41.734 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,8b70f0e747ab66e0,b04b87f080d6ef5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=8b70f0e747ab66e0, duration=9 ms] 2025-04-08 21:09:42.355 INFO 26087 --- [prometheus-minute-schedule-1,2ad3c3dcd29dd795,fa66f15064a8e017] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:42.573 WARN 26087 --- [ocp-async-3,1468c296a6a3cbca,ce453256ae3e0173] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:42.573 INFO 26087 --- [ocp-async-9,4a3c290cdda50d17,5386aae0733d63a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:44.211 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:44.211 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:44.573 WARN 26087 --- [ocp-async-9,64b15856c57e39d7,76481f449480f117] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:44.573 INFO 26087 --- [ocp-async-8,2591f7c814691a6d,0a7412edb8660262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:45.355 INFO 26087 --- [prometheus-minute-schedule-1,1aaa8892b3100c3c,71297266e0224779] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:46.521 INFO 26087 --- [ocp-schedules-15,d3ed6c2aa40c5904,ecb96cc218614a0a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:09:46.522 INFO 26087 --- [ocp-schedules-11,dd485bf03e583f67,f11de5e904811f3a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:09:46.522 INFO 26087 --- [ocp-schedules-11,dd485bf03e583f67,f11de5e904811f3a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:09:46.523 INFO 26087 --- [ocp-schedules-19,f25868d0dab50863,c0a1c4167ac2a425] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:09:46.523 INFO 26087 --- [ocp-schedules-19,f25868d0dab50863,c0a1c4167ac2a425] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:09:46.523 INFO 26087 --- [ocp-schedules-21,65c9bfebf04f7807,49af49c01100d9b1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:09:46.523 INFO 26087 --- [ocp-schedules-21,65c9bfebf04f7807,49af49c01100d9b1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:09:46.524 INFO 26087 --- [ocp-schedules-10,3092cf57fc48485d,b206b636f4dcc077] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:09:46.524 INFO 26087 --- [ocp-schedules-30,e429bfdf79ff0d0a,27893a13533b547b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:09:46.524 INFO 26087 --- [ocp-schedules-4,b0f39864a6a9ec77,1a9d368592d6d698] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:09:46.525 INFO 26087 --- [ocp-schedules-2,0fed3bc7c0708610,00341133ec061abf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:09:46.543 INFO 26087 --- [ocp-async-2,09ccc3ab0e383c94,d6d6654ebe3523ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:09:46.573 WARN 26087 --- [ocp-async-7,65d116203d3afc74,33333f916d38b222] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:46.574 INFO 26087 --- [ocp-async-6,9639b314c3a18ce3,d18e54a10d761a30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:46.586 INFO 26087 --- [ocp-schedules-22,0c53904d01fd2f36,e12103060a9e03a5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:09:46.602 INFO 26087 --- [ocp-schedules-8,53b375ebbeab3118,93ba2a9e92b9cb72] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:09:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:09:46.623517+08:00, lastProcessTime=2025-04-08T21:09:36.622759+08:00 2025-04-08 21:09:46.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:09:46.623517+08:00 2025-04-08 21:09:47.126 INFO 26087 --- [ocp-schedules-27,03f5ad47f24464fa,f6dfa2c6a13aeee9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:09:47.127 INFO 26087 --- [ocp-schedules-27,03f5ad47f24464fa,f6dfa2c6a13aeee9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:09:47.211 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:47.212 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:48.187 INFO 26087 --- [ocp-schedules-30,3408e8bc3b65b49a,1234dab17bac554c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:09:48.355 INFO 26087 --- [prometheus-minute-schedule-1,fa72ab89860cc64e,13e6853c8614c047] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:48.574 WARN 26087 --- [ocp-async-4,14372d0b64384eed,3d34ed5065041f98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:48.574 INFO 26087 --- [ocp-async-10,b52e80bb25f1101e,d14daeb39739cdaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:49.994 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e5cb813197750728,448bad6e77961905] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId e5cb813197750728] 2025-04-08 21:09:49.996 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e5cb813197750728,448bad6e77961905] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=e5cb813197750728, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.62, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.63, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.64, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.65, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 21:09:49.999 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e5cb813197750728,448bad6e77961905] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.62, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.63, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.64, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.65, sshPort=22, vpcId=1)] 2025-04-08 21:09:50.000 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 21:09:50.000 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 21:09:50.000 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 21:09:50.000 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] n.s.sshj.transport.random.JCERandom : Creating new SecureRandom. 2025-04-08 21:09:50.014 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 21:09:50.015 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 21:09:50.015 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 21:09:50.015 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] n.schmizz.sshj.transport.TransportImpl : Client identity string: SSH-2.0-SSHJ_0.34.0 2025-04-08 21:09:50.022 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 21:09:50.022 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 21:09:50.022 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 21:09:50.025 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] n.schmizz.sshj.transport.TransportImpl : Server identity string: SSH-2.0-OpenSSH_8.0 2025-04-08 21:09:50.033 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.62:22 success 2025-04-08 21:09:50.035 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.65:22 success 2025-04-08 21:09:50.037 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.64:22 success 2025-04-08 21:09:50.037 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshChannel : Connected ssh to host 192.168.1.63:22 success 2025-04-08 21:09:50.052 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@192.168.1.62 2025-04-08 21:09:50.052 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.62, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:4 2025-04-08 21:09:50.054 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@192.168.1.65 2025-04-08 21:09:50.054 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.65, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:5 2025-04-08 21:09:50.056 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@192.168.1.64 2025-04-08 21:09:50.056 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.64, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:6 2025-04-08 21:09:50.073 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshChannel : Success to init SSHClient by PASSWORD, host: root@192.168.1.63 2025-04-08 21:09:50.073 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=ssh, hostAddress=192.168.1.63, httpPort=null, sshPort=22, authentication=Authentication(httpAuth=null, sshAuth=SshAuthentication(authType=password, passwordAuthConfig=PasswordAuthConfig(username=root), privateKeyAuthConfig=null)), proxy=null), estimated size:7 2025-04-08 21:09:50.101 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:50.103 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:50.105 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:50.117 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.118 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.122 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:50.122 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.138 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.158 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:50.160 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:50.162 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:50.173 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.173 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.173 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:50.178 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.178 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.178 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:50.179 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.179 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.179 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:50.179 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:50.188 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.188 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.65 2025-04-08 21:09:50.195 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.195 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.64 2025-04-08 21:09:50.195 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.196 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.62 2025-04-08 21:09:50.197 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.197 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.197 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:50.212 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:50.213 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:50.214 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.214 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.214 INFO 26087 --- [pool-24-thread-4,e5cb813197750728,672be3dba5823eaa] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.214 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.63 2025-04-08 21:09:50.226 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.226 INFO 26087 --- [pool-24-thread-3,e5cb813197750728,83feb0c774f5f636] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.228 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.228 INFO 26087 --- [pool-24-thread-1,e5cb813197750728,2dafd8da782452fc] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.244 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.244 INFO 26087 --- [pool-24-thread-2,e5cb813197750728,50c5daccd03bb1b6] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:50.244 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e5cb813197750728,448bad6e77961905] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=e5cb813197750728, duration=252 ms] 2025-04-08 21:09:50.574 WARN 26087 --- [ocp-async-1,1e2332a1bb16dfd2,becdb811001bbb8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:50.574 INFO 26087 --- [ocp-async-9,ada037a17c3ccf47,3b865e06e83d5e31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:51.355 INFO 26087 --- [prometheus-minute-schedule-1,9b265b6daefd3b9d,660e9f9d21f648b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:52.574 WARN 26087 --- [ocp-async-2,12354e360e96c6e0,6a97a9c189f1e7f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:52.574 INFO 26087 --- [ocp-async-5,bb4fd44aec411dc4,dc32e59db35b31c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:53.213 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:53.214 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:54.355 INFO 26087 --- [prometheus-minute-schedule-1,0a7e8d98a8612ef3,03204a969dd8bb84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:54.574 WARN 26087 --- [ocp-async-4,8209ff4ab308df90,a89dd050b6c90411] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:54.574 INFO 26087 --- [ocp-async-3,c0731b543391b290,5ed22ffb3e7f4fd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:55.475 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,30a4c2e40984b94a,2940781514a760f0] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId 30a4c2e40984b94a] 2025-04-08 21:09:55.477 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,30a4c2e40984b94a,2940781514a760f0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=30a4c2e40984b94a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.62, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.63, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.64, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.65, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 21:09:55.480 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,30a4c2e40984b94a,2940781514a760f0] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.62, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.63, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.64, sshPort=22, vpcId=1), BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.65, sshPort=22, vpcId=1)] 2025-04-08 21:09:55.481 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:55.481 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:55.481 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:55.483 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:55.497 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.497 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.498 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:55.498 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.498 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:55.501 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.512 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.512 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:55.514 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.515 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:55.527 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.527 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.527 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:55.532 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.532 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.537 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:55.541 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.542 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.65 2025-04-08 21:09:55.543 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:55.554 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.555 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:55.561 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.561 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:55.568 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.568 INFO 26087 --- [pool-25-thread-4,30a4c2e40984b94a,bf3434897f3af9d2] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.571 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.571 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.573 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:55.578 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.578 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.578 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:55.589 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.589 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.63 2025-04-08 21:09:55.594 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.594 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.62 2025-04-08 21:09:55.611 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:55.619 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.619 INFO 26087 --- [pool-25-thread-2,30a4c2e40984b94a,fa4c72bb7eb73d36] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.625 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.625 INFO 26087 --- [pool-25-thread-1,30a4c2e40984b94a,2dda148216bbf6c7] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.627 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.627 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.64 2025-04-08 21:09:55.654 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.654 INFO 26087 --- [pool-25-thread-3,30a4c2e40984b94a,5e324d755708591f] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:55.654 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,30a4c2e40984b94a,2940781514a760f0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=30a4c2e40984b94a, duration=181 ms] 2025-04-08 21:09:55.840 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOver, client 10.11.12.102, traceId 2eb36642664461ac] 2025-04-08 21:09:55.842 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOver?null, client=10.11.12.102, traceId=2eb36642664461ac, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster(TakeOverClusterParam), args=TakeOverClusterParam(switchConfigUrl=true, hostInfo=TakeOverClusterParam.HostInfo(kind=DEDICATED_PHYSICAL_MACHINE, publishPorts=null, hostTypeId=1, sshPort=22, credentialId=1)),] 2025-04-08 21:09:55.851 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-192.168.1.62#2881, method=takeOverCluster 2025-04-08 21:09:55.855 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.855 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.859 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.859 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.863 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.863 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.867 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.868 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.871 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.872 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.875 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.875 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 21:09:55.888 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.889 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.892 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.893 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 21:09:55.905 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.905 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.909 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.909 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 21:09:55.924 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.924 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.929 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.929 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.933 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.933 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 21:09:55.934 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.934 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.937 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.938 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.941 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.942 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 21:09:55.954 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.954 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 21:09:55.959 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.960 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.963 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.965 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 21:09:55.967 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.968 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.972 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.972 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.975 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:09:55.975 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.979 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:09:55.993 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.994 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:55.997 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:55.998 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:09:56.048 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.049 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.053 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.054 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.058 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.059 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.063 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.063 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.067 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.068 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:09:56.122 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.123 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.127 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.128 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:09:56.130 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.130 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.134 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.135 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:09:56.191 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.192 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.197 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.197 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.201 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.202 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.206 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.206 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.210 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.210 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:09:56.214 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:56.214 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:56.264 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.265 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.269 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.269 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:09:56.273 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.274 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.279 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.279 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:09:56.332 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.333 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.336 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.337 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.341 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.341 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.345 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.346 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.350 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.351 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:09:56.408 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.409 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.414 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.415 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:09:56.420 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.420 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.424 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.425 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.429 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.429 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.432 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.433 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 21:09:56.462 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.463 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT name, value FROM `ocp`.`ops_config` WHERE name = ?, args: [ob.proxyro.password] 2025-04-08 21:09:56.463 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.464 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.467 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.468 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:56.473 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.473 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:56.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:09:56.478 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.478 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.482 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.483 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.486 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.487 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.490 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.490 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.494 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.494 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:09:56.506 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.507 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.511 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.512 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.516 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.519 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `user`, (CASE `account_locked` WHEN 'Y' THEN 1 ELSE 0 END) AS account_locked, `select_priv`, `insert_priv`, `update_priv`, `delete_priv`, `create_priv`, `drop_priv`, `process_priv`, `grant_priv`, `index_priv`, `alter_priv`, `show_db_priv`, `super_priv`, `create_view_priv`, `show_view_priv`, `create_user_priv`, `create_routine_priv`, `alter_routine_priv`, `execute_priv`, `password`, `host` FROM `mysql`.`user` WHERE `user` = binary ? AND `host` = ?, args: [proxyro, %] 2025-04-08 21:09:56.520 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.520 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW DATABASES 2025-04-08 21:09:56.524 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.524 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.529 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.530 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `db`, `user`, `select_priv`, `insert_priv`, `update_priv`, `delete_priv`, `create_priv`, `drop_priv`, `index_priv`, `alter_priv`, `create_view_priv`, `show_view_priv` , `host` FROM `mysql`.`db` WHERE `user` = binary ? AND `host` = ?, args: [proxyro, %] 2025-04-08 21:09:56.533 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.62] 2025-04-08 21:09:56.536 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.63] 2025-04-08 21:09:56.538 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.64] 2025-04-08 21:09:56.540 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.65] 2025-04-08 21:09:56.543 INFO 26087 --- [ocp-async-3,a50477d2ee9fe339,f463f40b12831517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:09:56.547 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.host.service.RegionServiceImpl : Created region: 1000003 sys_region 2025-04-08 21:09:56.552 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.host.service.IdcServiceImpl : Created idc: 1000002 default_idc 2025-04-08 21:09:56.559 INFO 26087 --- [ocp-schedules-24,76a6bdb441989c29,b291ae8c909c7233] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:09:56.574 WARN 26087 --- [ocp-async-8,8cbd21948c963cca,e74420ac7fc97232] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:56.574 INFO 26087 --- [ocp-async-9,e7c725b6918582cb,e611b682c88e2d5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:56.582 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000002, resourceId=1000002 2025-04-08 21:09:56.584 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000003, resourceId=1000003 2025-04-08 21:09:56.595 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000004, resourceId=1000004 2025-04-08 21:09:56.597 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000005, resourceId=1000005 2025-04-08 21:09:56.603 INFO 26087 --- [ocp-schedules-23,595665ce08f16b04,d12f17c784e45e80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:09:56.606 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.o.i.c.ClusterGroupServiceImpl : create cluster group, name=sgkydb, description=sgkydb, create by cluster service, affectedRows=1 2025-04-08 21:09:56.613 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000002, resourceId=1000002 2025-04-08 21:09:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:09:56.623132+08:00, lastProcessTime=2025-04-08T21:09:46.623517+08:00 2025-04-08 21:09:56.626 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000002, tenantId=null, clusterName=sgkydb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:09:56.633 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000002, tenantId=null, clusterName=sgkydb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:09:56.634 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000002, tenantId=null, clusterName=sgkydb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:09:56.635 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.CredentialServiceImpl : Credential clusterName=sgkydb|tenantName=sys|username=root is already exists 2025-04-08 21:09:56.635 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.CredentialServiceImpl : Delete credential:clusterName=sgkydb|tenantName=sys|username=root 2025-04-08 21:09:56.638 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000002, tenantId=null, clusterName=sgkydb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:09:56.639 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.639 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:56.639 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.640 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:09:56.640 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.640 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:09:56.644 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.645 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.649 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.649 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.652 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.653 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.657 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.657 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:09:56.661 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:09:56.662 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:09:56.699 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE APP GROUP, property:UpdateAppGroupProperty(serialNumbers=[null, null, null, null], appGroup=null) 2025-04-08 21:09:56.700 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000008, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:09:56.700 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Take over primary OB cluster, nodeCount=39, taskCount=73 2025-04-08 21:09:56.700 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000002, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, ssh_port=22, prohibit_rollback=false, service_name=sgkydb:1744016068, ob_connect_address=192.168.1.62:2881, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, createTime=null, startTime=2025-04-08T21:09:56.700+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=12, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Create random agent credential, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=35, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Uninstall legacy ocp agent, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UninstallLegacyOcpAgentTask, seriesId=57, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Prepare take over cluster, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask, seriesId=72, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Attach cluster attributes, state=PENDING, operation=EXECUTE, classN 2025-04-08 21:09:56.732 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:09:56.623132+08:00 2025-04-08 21:09:56.761 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000091} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.768 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000054} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.768 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000111} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.771 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000102} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.773 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000067} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.775 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000089} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.777 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000062} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000004, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.778 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000072} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000003, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.779 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000083} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.779 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000081} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.781 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000076} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.782 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000113} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000002, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.783 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000086} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000004, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.784 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000066} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.785 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000061} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.786 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000060} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.786 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000096} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.787 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000097} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000002, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.788 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000063} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.788 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000071} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.789 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000055} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.790 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000057} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone2', tenantName='null', tenantId=null, obTenantId=null, serverId=1000003, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.791 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000112} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.791 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000073} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.792 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000088} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.793 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000098} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.794 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000100} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.795 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000109} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000005, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.796 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000045} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.796 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000103} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.797 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000115} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.798 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000077} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000046} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.799 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000049} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.800 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000084} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.801 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000105} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.802 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000116} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.803 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000078} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000002, svrIp='192.168.1.64', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.803 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000110} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000003, svrIp='192.168.1.65', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.805 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000079} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.805 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000099} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.806 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000101} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.807 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000087} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.808 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000108} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000004, svrIp='192.168.1.62', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.809 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000044} BaseTagEntity{id=null, clusterName='sgkydb', clusterId=1000002, obClusterId=1744016068, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000005, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.809 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000069} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000005, svrIp='192.168.1.63', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:09:56.810 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=1000007, taskName=Take over primary OB cluster, argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, ssh_port=22, prohibit_rollback=false, service_name=sgkydb:1744016068, ob_connect_address=192.168.1.62:2881, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}} 2025-04-08 21:09:56.833 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOver client=10.11.12.102, traceId=2eb36642664461ac, duration=995 ms] 2025-04-08 21:09:56.833 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2eb36642664461ac,952dcd42b5a30d41] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.62:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=1000007, eventInfo={"param":{"address":"192.168.1.62","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T21:09:55.842+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 21:09:56.898 INFO 26087 --- [ocp-schedules-10,783b684e1cc665bb,1643f03973c97292] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:09:56.907 INFO 26087 --- [ocp-schedules-10,783b684e1cc665bb,1643f03973c97292] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:09:56.907 INFO 26087 --- [ocp-schedules-10,783b684e1cc665bb,1643f03973c97292] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:09:57.135 INFO 26087 --- [ocp-schedules-28,1930d349015b4195,b7dd71e243b22966] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:09:57.137 INFO 26087 --- [ocp-schedules-28,1930d349015b4195,b7dd71e243b22966] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:09:57.355 INFO 26087 --- [prometheus-minute-schedule-1,1e2a8ab5d6e1b84b,e7f4fc8c47917909] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117800, interval=60 2025-04-08 21:09:57.415 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,d6242879d531ad38] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000047, operation:EXECUTE, state: RUNNING 2025-04-08 21:09:57.425 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bb4fa5d858ca14cf] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:09:57.434 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bb4fa5d858ca14cf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000047, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, ssh_port=22, service_name=sgkydb:1744016068, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, latest_execution_start_time=2025-04-08T21:09:57.411+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=1000047}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:09:57.442 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bb4fa5d858ca14cf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000047, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:09:57.448 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bb4fa5d858ca14cf] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000063, 1000078, 1000101, 1000110] 2025-04-08 21:09:57.734 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [sgkydb] cache 2025-04-08 21:09:57.734 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:09:57.734 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster sgkydb cache info from db 2025-04-08 21:09:57.736 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000008, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:09:57.736 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster sgkydb cache info from db success. 2025-04-08 21:09:58.205 INFO 26087 --- [ocp-schedules-21,eff0678855109f7d,cff98fcf0638d053] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:09:58.374 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000008 2025-04-08 21:09:58.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,8341bedf90459ac5,268bf0de9c9ccf76] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=8341bedf90459ac5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:09:58.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,ff9d0a0bf9fd5f0f,d141cee00c2e4631] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=ff9d0a0bf9fd5f0f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:09:58.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,8341bedf90459ac5,268bf0de9c9ccf76] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=8341bedf90459ac5, duration=3 ms] 2025-04-08 21:09:58.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,fb58ebbef4f82771,b6736a0a5da41060] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=fb58ebbef4f82771, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:09:58.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,1281de27d40f07b1,1f8e19bac862f43b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=1281de27d40f07b1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:09:58.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,fb58ebbef4f82771,b6736a0a5da41060] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=fb58ebbef4f82771, duration=3 ms] 2025-04-08 21:09:58.385 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,c59d357783765bb1,1c4743e486a718b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=c59d357783765bb1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:09:58.387 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,ff9d0a0bf9fd5f0f,d141cee00c2e4631] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=ff9d0a0bf9fd5f0f, duration=8 ms] 2025-04-08 21:09:58.403 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,1281de27d40f07b1,1f8e19bac862f43b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=1281de27d40f07b1, duration=24 ms] 2025-04-08 21:09:58.423 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,cd6f29913163b30c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000063, operation:EXECUTE, state: RUNNING 2025-04-08 21:09:58.436 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,cd6f29913163b30c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000078, operation:EXECUTE, state: RUNNING 2025-04-08 21:09:58.446 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:09:58.461 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,cd6f29913163b30c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000101, operation:EXECUTE, state: RUNNING 2025-04-08 21:09:58.458 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000063, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:09:58.421+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000063}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:09:58.463 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:09:58.472 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,cd6f29913163b30c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000110, operation:EXECUTE, state: RUNNING 2025-04-08 21:09:58.479 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,f459c80c27f8514f,bb7043d5c3e7765c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=f459c80c27f8514f, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:09:58.486 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,f459c80c27f8514f,bb7043d5c3e7765c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=f459c80c27f8514f, duration=10 ms] 2025-04-08 21:09:58.471 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000078, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:09:58.435+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000078}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:09:58.488 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,c59d357783765bb1,1c4743e486a718b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=c59d357783765bb1, duration=109 ms] 2025-04-08 21:09:58.481 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:09:58.487 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:58.503 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:58.513 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000101, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:09:58.459+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000101}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:09:58.515 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:09:58.539 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:09:58.539 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:09:58.539 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:09:58.535 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.543 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.62 2025-04-08 21:09:58.537 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000110, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:09:58.471+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000110}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:09:58.542 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.575 WARN 26087 --- [ocp-async-7,5c526ef7c2bfd6bb,bdd0e51d99c77382] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:09:58.575 INFO 26087 --- [ocp-async-5,836e973cb92b2fe7,6e9f5638d5bc37d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:09:58.545 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:58.551 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:58.563 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.64 2025-04-08 21:09:58.572 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:58.611 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.612 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:58.621 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.627 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.634 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.63 2025-04-08 21:09:58.636 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.637 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.65 2025-04-08 21:09:58.647 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.64 2025-04-08 21:09:58.652 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.62 2025-04-08 21:09:58.662 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:58.668 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:58.704 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.720 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.723 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 21:09:58.724 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.727 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.730 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.734 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.64, port=22, loopCount=1, minSuccessCount=1 2025-04-08 21:09:58.737 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.744 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.63 2025-04-08 21:09:58.757 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.65 2025-04-08 21:09:58.762 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:09:58.794 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 21:09:58.808 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.62, port=22, loopCount=1, minSuccessCount=1 2025-04-08 21:09:58.811 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:09:58.817 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.820 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.820 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.822 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.831 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.833 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.834 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 21:09:58.837 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.64 2025-04-08 21:09:58.844 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 21:09:58.848 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.63, port=22, loopCount=1, minSuccessCount=1 2025-04-08 21:09:58.851 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.65, port=22, loopCount=1, minSuccessCount=1 2025-04-08 21:09:58.858 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:09:58.860 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:09:58.865 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.867 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.869 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[108] 2025-04-08 21:09:58.870 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=108, maxAcceptableDelayMillis=3000 2025-04-08 21:09:58.871 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.64], timeoutMillis=20000 2025-04-08 21:09:58.874 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.62 2025-04-08 21:09:58.875 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.877 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.883 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744117798 0 0 , error= 2025-04-08 21:09:58.890 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.63 2025-04-08 21:09:58.892 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.65 2025-04-08 21:09:58.893 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 21:09:58.900 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.902 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ee3600c7d67ef2d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000078, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:09:58.903 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.905 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[94] 2025-04-08 21:09:58.910 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.912 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=94, maxAcceptableDelayMillis=3000 2025-04-08 21:09:58.912 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.915 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.920 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.62], timeoutMillis=20000 2025-04-08 21:09:58.935 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:09:58.937 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[85] 2025-04-08 21:09:58.951 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744117798 0 0 , error= 2025-04-08 21:09:58.957 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[103] 2025-04-08 21:09:58.966 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=85, maxAcceptableDelayMillis=3000 2025-04-08 21:09:58.970 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 21:09:58.994 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=103, maxAcceptableDelayMillis=3000 2025-04-08 21:09:59.002 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.63], timeoutMillis=20000 2025-04-08 21:09:59.011 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.65], timeoutMillis=20000 2025-04-08 21:09:59.015 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,f00d9749a6a9fc6a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000063, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:09:59.024 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744117799 0 0 , error= 2025-04-08 21:09:59.026 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 21:09:59.030 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,44cfca3268d478e9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000110, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:09:59.215 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:09:59.215 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:09:59.327 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,bb387529d9ed0866,06b49d240f821839] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007/subtasks/1000063/logSegments?taskInstanceId=1000007&subtaskInstanceId=1000063, client=10.11.12.102, traceId=bb387529d9ed0866, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000007,1000063,] 2025-04-08 21:09:59.329 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,bb387529d9ed0866,06b49d240f821839] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007/subtasks/1000063/logSegments client=10.11.12.102, traceId=bb387529d9ed0866, duration=6 ms] 2025-04-08 21:09:59.957 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744117799 0 0 , error= 2025-04-08 21:10:00.000 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 21:10:00.006 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000101, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:00.008 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,2828164aaa7ec5cf] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000064] 2025-04-08 21:10:00.355 INFO 26087 --- [prometheus-minute-schedule-1,e53ce8946af19856,99f2a6e728964d1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:00.386 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,48947bcbad9d5601,2141e0a55d1d091c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=48947bcbad9d5601, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:00.402 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,48947bcbad9d5601,2141e0a55d1d091c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=48947bcbad9d5601, duration=21 ms] 2025-04-08 21:10:00.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,26dd8f7537eb649e,efadc16c01049107] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=26dd8f7537eb649e, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:10:00.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,fc0d28cc32a63f00,b407cf41bb51d927] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=fc0d28cc32a63f00, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:10:00.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,c518b5dce553e7be,edea878841b5bda6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=c518b5dce553e7be, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:10:00.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,fc0d28cc32a63f00,b407cf41bb51d927] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=fc0d28cc32a63f00, duration=4 ms] 2025-04-08 21:10:00.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,26dd8f7537eb649e,efadc16c01049107] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=26dd8f7537eb649e, duration=4 ms] 2025-04-08 21:10:00.442 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,c518b5dce553e7be,edea878841b5bda6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=c518b5dce553e7be, duration=4 ms] 2025-04-08 21:10:00.464 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:10:00.464 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:10:00.464 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:10:00.495 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,6faa5d5f397a92d4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000064, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:00.530 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,99c7fd3345e42055] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:00.538 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,99c7fd3345e42055] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000064, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:00.494+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000064}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:00.559 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,99c7fd3345e42055] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000064, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:00.559 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:05:00.0, nextRun=2025-04-08 21:10:00.0, currentDate=Tue Apr 08 21:10:00 CST 2025 2025-04-08 21:10:00.559 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:05:00.0, nextRunTime=2025-04-08 21:10:00.0} 2025-04-08 21:10:00.561 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,99c7fd3345e42055] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000066, 1000081, 1000096, 1000103] 2025-04-08 21:10:00.563 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:10:00.563 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:10:00.563+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:10:00.575 WARN 26087 --- [ocp-async-4,79d473b9e3bd7c5b,761a4aaeb875ecb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:00.575 INFO 26087 --- [ocp-async-6,874d13a00780a8bf,464e17a5daeca088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:00.597 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:05:00.0, nextRun=2025-04-08 21:10:00.0, currentDate=Tue Apr 08 21:10:00 CST 2025 2025-04-08 21:10:00.597 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:05:00.0, nextRunTime=2025-04-08 21:10:00.0} 2025-04-08 21:10:00.605 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:10:00.607 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:10:00.607 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,e52788379b7a185a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:10:00.819 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,d23f5844c6703751,b7ee6842867e8d6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007/subtasks/1000063/logSegments?taskInstanceId=1000007&subtaskInstanceId=1000063, client=10.11.12.102, traceId=d23f5844c6703751, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000007,1000063,] 2025-04-08 21:10:00.822 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,d23f5844c6703751,b7ee6842867e8d6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007/subtasks/1000063/logSegments client=10.11.12.102, traceId=d23f5844c6703751, duration=8 ms] 2025-04-08 21:10:01.521 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b8acc71145e91d1e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000066, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:01.532 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b8acc71145e91d1e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000081, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:01.543 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b8acc71145e91d1e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000096, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:01.541 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:01.563 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b8acc71145e91d1e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000103, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:01.549 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:01.595 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,b8acc71145e91d1e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000117, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:01.560 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000066, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:01.519+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000066}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:01.596 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 21:10:01.572 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000081, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:01.530+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000081}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:01.651 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 21:10:01.594 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:01.606 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:01.610 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:01.655 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:10:01.659 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000096, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:01.541+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000096}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:01.661 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:01.691 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000103, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:01.562+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000103}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:01.693 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000117, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000008, task_operation=execute, latest_execution_start_time=2025-04-08T21:10:01.593+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000117}, listMap={}}, executor=192.168.1.61 2025-04-08 21:10:01.702 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 21:10:01.710 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:01.717 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 21:10:01.817 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,eb95c8556766dccd,94a3d94b8bab10aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=eb95c8556766dccd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:01.832 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,eb95c8556766dccd,94a3d94b8bab10aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=eb95c8556766dccd, duration=19 ms] 2025-04-08 21:10:01.722 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:01.725 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:01.728 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:01.751 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:01.795 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:01.833 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:01.834 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for sgkydb 2025-04-08 21:10:01.867 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:01.941 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:01.948 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:02.005 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:01.954 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:01.964 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:01.977 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:02.006 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:02.009 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.62 2025-04-08 21:10:02.040 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:02.056 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.64 2025-04-08 21:10:02.064 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:02.079 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.099 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:02.101 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:02.101 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:10:02.105 INFO 26087 --- [subtask-executor5,8ec1a44aeefd3a62,5ac8b39146fe6f23] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000117, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:02.111 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.119 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.63 2025-04-08 21:10:02.122 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.128 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.152 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.65 2025-04-08 21:10:02.179 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.196 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.197 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.215 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:02.216 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:02.213 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.220 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 21:10:02.220 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 21:10:02.232 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.238 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,350a99381e460ab0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000081, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:02.240 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:02.238 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 21:10:02.246 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,1f7f5ca5d96a14cc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000066, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:02.296 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 21:10:02.313 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,bd68f7338cc82da5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000096, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:02.321 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000103, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:02.323 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5c198e275e33b9ea] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000092] 2025-04-08 21:10:02.549 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000008, status:SUCCESS 2025-04-08 21:10:02.549 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000008, status:SUCCESS 2025-04-08 21:10:02.575 WARN 26087 --- [ocp-async-1,c2a5a9a9e7a46df3,54a30570235f6341] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:02.575 INFO 26087 --- [ocp-async-10,dd943896700b383e,236202b47ccf481e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:02.610 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,e473026ef21b25b5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000092, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:02.621 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,2deb5f48e7f8afc0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:02.638 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,2deb5f48e7f8afc0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000092, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:02.608+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000092}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:02.646 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,2deb5f48e7f8afc0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000092, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:02.648 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,2deb5f48e7f8afc0] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000049, 1000055, 1000089, 1000099] 2025-04-08 21:10:03.187 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,5d8c61a660b063e2,fc39a24db0e250e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=5d8c61a660b063e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:03.202 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,5d8c61a660b063e2,fc39a24db0e250e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=5d8c61a660b063e2, duration=19 ms] 2025-04-08 21:10:03.355 INFO 26087 --- [prometheus-minute-schedule-1,5ea6999ce8381eeb,49f906a682d4d1d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:03.439 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,dd8c3e691b252e77,63ed3025ddf6c5db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=dd8c3e691b252e77, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:10:03.441 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,dd8c3e691b252e77,63ed3025ddf6c5db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=dd8c3e691b252e77, duration=5 ms] 2025-04-08 21:10:03.621 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,47f83cf5f46aaea9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000049, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:03.652 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,47f83cf5f46aaea9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000055, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:03.687 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,47f83cf5f46aaea9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000089, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:03.665 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:03.690 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000049, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:03.619+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000049}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:03.701 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 21:10:03.705 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,47f83cf5f46aaea9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000099, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:03.704 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:03.708 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:03.710 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:03.716 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000055, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:03.651+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000055}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:03.723 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000089, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:03.686+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000089}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:03.729 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:03.741 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 21:10:03.744 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 21:10:03.753 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000099, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:03.704+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000099}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:03.753 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.778 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:03.797 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:03.805 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 21:10:03.810 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:03.824 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:03.827 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.828 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.838 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.841 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:03.848 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.853 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:03.868 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:03.886 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.893 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:03.932 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.937 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:03.939 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.962 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.963 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:03.994 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:04.000 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.006 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:04.010 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:04.018 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.039 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.046 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.051 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:04.066 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.113 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.075 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.093 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.64 2025-04-08 21:10:04.117 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:04.120 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:04.125 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.152 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.154 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.167 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.62 2025-04-08 21:10:04.168 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.63 2025-04-08 21:10:04.194 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.65 2025-04-08 21:10:04.479 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.492 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.496 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 21:10:04.521 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,e9ef5f39847ba52d,3b5096d6a39a4313] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=e9ef5f39847ba52d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:04.503 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 21:10:04.537 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,e9ef5f39847ba52d,3b5096d6a39a4313] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=e9ef5f39847ba52d, duration=20 ms] 2025-04-08 21:10:04.575 WARN 26087 --- [ocp-async-7,665b1bcf4397bd6b,de3c90fb19c15e48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:04.575 INFO 26087 --- [ocp-async-9,1c3b7b5a2ed83671,12677578065cafad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:04.534 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:04.562 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.572 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.591 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.597 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.626 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.630 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.638 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.668 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.64 2025-04-08 21:10:04.724 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,590021e0f80909b8] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:03:27.0, nextRun=2025-04-08 21:10:04.0, currentDate=Tue Apr 08 21:10:04 CST 2025 2025-04-08 21:10:04.724 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,590021e0f80909b8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-04-08 21:03:27.0, nextRunTime=2025-04-08 21:10:04.0} 2025-04-08 21:10:04.727 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,590021e0f80909b8] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 21:10:04.727 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,590021e0f80909b8] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:10:04.727+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:10:04.711 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 21:10:04.715 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 21:10:04.718 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 21:10:04.761 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 21:10:04.774 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 21:10:04.799 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 21:10:04.806 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:04.810 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:04.815 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:04.835 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.839 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.846 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:04.851 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.63 2025-04-08 21:10:04.857 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.65 2025-04-08 21:10:04.877 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.62 2025-04-08 21:10:05.085 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.091 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.099 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 21:10:05.108 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 21:10:05.113 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 21:10:05.123 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,711e99fc73d06219] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000049, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:05.216 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:05.217 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:05.217 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.219 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.220 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.224 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 21:10:05.233 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.240 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.242 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 21:10:05.249 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:05.258 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 21:10:05.260 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 21:10:05.266 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 21:10:05.276 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 21:10:05.298 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 21:10:05.299 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,10c015e44ddddcab] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000089, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:05.307 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 21:10:05.310 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 21:10:05.322 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,2b11990b3730efa1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000099, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:05.333 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000055, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:05.335 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e02a2b312adb5b10] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000050] 2025-04-08 21:10:05.724 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,69946fe56e681073] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000050, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:05.755 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,69946fe56e681073] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000118, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:05.764 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,a24d8c75d371e61b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:05.793 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,a24d8c75d371e61b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000050, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:05.722+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000050}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:05.810 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:10:05.813 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000118, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000009, task_operation=execute, latest_execution_start_time=2025-04-08T21:10:05.753+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=1000118}, listMap={}}, executor=192.168.1.61 2025-04-08 21:10:05.816 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,a24d8c75d371e61b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000050, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:05.817 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,a24d8c75d371e61b] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000046, 1000073, 1000088, 1000098] 2025-04-08 21:10:05.819 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:10:05.819 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:10:05.819 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.820 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:10:05.821 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.821 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.821 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.822 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:10:05.833 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.834 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.834 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.834 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:10:05.845 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.845 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.846 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.846 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:10:05.856 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.857 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.857 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.857 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:10:05.875 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.876 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.876 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.876 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:10:05.882 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.882 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.882 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.882 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:10:05.886 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.886 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.886 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.887 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:10:05.890 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.891 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.891 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.891 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:10:05.896 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.896 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.896 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.897 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:10:05.901 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.901 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.901 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.901 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:10:05.905 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.906 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.906 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.906 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:10:05.913 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.913 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.913 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.914 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:10:05.925 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.925 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.926 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.926 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:10:05.926 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,79d313014acc6016,fb255797cc31b19c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=79d313014acc6016, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:05.930 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.930 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.930 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.931 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:10:05.935 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.935 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.936 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.936 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:10:05.940 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.940 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.941 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.941 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:10:05.942 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,79d313014acc6016,fb255797cc31b19c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=79d313014acc6016, duration=20 ms] 2025-04-08 21:10:05.945 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.945 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.946 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.946 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:10:05.949 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.950 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.950 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.950 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:10:05.953 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.954 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.954 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.954 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:10:05.958 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.958 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.959 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.959 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:10:05.962 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.962 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.963 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.963 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:10:05.966 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.967 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.967 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.967 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:10:05.971 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.971 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.971 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.971 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:10:05.975 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.975 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.975 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.976 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:10:05.979 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.979 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.980 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.980 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:10:05.983 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.983 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.984 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.984 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:10:05.987 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.988 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.988 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.988 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:10:05.991 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.991 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.992 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.992 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:10:05.996 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.996 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:05.996 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:05.996 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:10:06.001 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.001 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.001 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.001 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:10:06.005 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.005 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.005 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.005 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:10:06.009 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.009 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.009 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.010 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:10:06.013 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.013 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.013 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.014 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:10:06.017 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.017 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.017 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.018 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:10:06.024 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.024 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.024 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.024 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:10:06.030 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.030 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.030 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.031 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:10:06.035 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.036 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.036 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.036 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:10:06.040 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.041 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.041 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.041 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:10:06.044 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.044 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.045 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.045 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:10:06.048 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.048 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.048 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.048 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:10:06.051 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.051 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.052 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.052 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:10:06.055 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.055 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.055 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.055 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:10:06.063 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.063 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.063 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.064 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:10:06.080 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.080 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.080 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.080 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:10:06.086 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.086 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.086 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.086 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:10:06.091 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.092 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.092 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.092 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:10:06.097 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.098 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.098 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.098 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:10:06.102 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.102 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.102 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.102 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:10:06.106 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.107 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.107 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.107 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:10:06.111 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:10:06.111 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:10:06.111 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.111 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:10:06.112 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.112 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.112 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.113 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:10:06.117 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.117 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:10:06.117 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:06.117 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:10:06.120 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:10:06.127 INFO 26087 --- [subtask-executor6,8ec1a44aeefd3a62,7658851d5da619ee] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000118, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:06.299 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:06:06.259, end=2025-04-08T21:08:06.292} 2025-04-08 21:10:06.303 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:10:06.303 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:10:06.355 INFO 26087 --- [prometheus-minute-schedule-1,881b434a533a1f35,c2fcdbf1d948988c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:06.355 INFO 26087 --- [prometheus-cache-monitor1,a8718c52c9dbfe7e,0b44dab35f85dd53] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1478,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":372,"seriesIdKeySize":423,"secondWriteQueue":0,"minuteWriteCacheSize":372} 2025-04-08 21:10:06.355 INFO 26087 --- [prometheus-cache-monitor1,a8718c52c9dbfe7e,0b44dab35f85dd53] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=7, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=6, cluster_cgroup_value=4, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:10:06.544 INFO 26087 --- [ocp-async-9,cbf0360770969526,6c8db3edf91df490] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:10:06.569 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000009, status:SUCCESS 2025-04-08 21:10:06.570 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000009, status:SUCCESS 2025-04-08 21:10:06.575 WARN 26087 --- [ocp-async-4,1f091942e090f528,359550de505d029c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:06.575 INFO 26087 --- [ocp-async-5,6d92e21adcf8e52c,d76cccf5e56376fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:06.604 INFO 26087 --- [ocp-schedules-19,62b56b5a53598c78,c9ac83a8e7ddac44] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:10:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:10:06.624311+08:00, lastProcessTime=2025-04-08T21:09:56.623132+08:00 2025-04-08 21:10:06.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-sgkydb, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 21:10:06.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:10:06.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-sgkydb, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 21:10:06.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:10:06.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:10:06.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:10:06.624311+08:00 2025-04-08 21:10:06.779 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,efea84df6c9df492] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000046, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:06.814 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,efea84df6c9df492] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000073, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:06.816 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:06.820 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,efea84df6c9df492] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000088, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:06.819 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000046, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:06.778+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000046}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:06.828 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,efea84df6c9df492] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000098, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:06.823 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 21:10:06.823 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:06.840 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:06.842 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:06.843 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000073, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:06.812+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000073}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:06.850 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:06.864 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:06.869 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000088, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:06.819+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000088}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:06.893 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 21:10:06.919 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000098, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:06.827+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000098}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:06.924 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:06.940 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 21:10:06.948 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:06.949 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 21:10:06.974 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:06.977 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.006 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.009 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.016 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.018 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.023 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.028 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.035 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.039 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.049 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.052 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.056 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:07.058 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.065 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:07.076 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:07.080 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.093 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.099 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:07.120 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.142 INFO 26087 --- [ocp-schedules-11,a081320525245a95,b2c00361278db330] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:10:07.143 INFO 26087 --- [ocp-schedules-11,a081320525245a95,b2c00361278db330] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:10:07.121 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.126 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:07.139 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.155 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.165 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.166 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.172 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.178 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.180 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.183 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:07.184 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.190 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.191 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.210 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.216 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.224 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:07.231 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:07.232 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:07.257 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.263 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,1ecab1ce62701245,3d8b3f663dcd428b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=1ecab1ce62701245, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:07.266 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.267 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.280 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,1ecab1ce62701245,3d8b3f663dcd428b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=1ecab1ce62701245, duration=21 ms] 2025-04-08 21:10:07.273 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.274 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.277 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:07.289 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.293 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.295 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.299 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.303 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.305 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:07.316 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.322 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.324 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.328 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.357 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:07.359 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.362 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:07.364 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.372 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:07.397 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.400 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.402 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.400 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:07.402 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.402 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.404 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.410 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.412 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.416 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.419 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:07.431 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.434 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.437 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.438 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.443 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.472 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:07.478 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:07.479 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.493 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:07.544 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.545 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.546 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.548 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.550 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.554 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:07.593 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.599 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.602 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.610 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.617 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:07.629 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.635 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.641 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.643 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.647 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.670 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:07.681 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:07.692 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:07.718 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.756 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.768 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.769 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.770 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.790 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.794 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.808 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.813 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:07.815 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.834 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.835 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:07.869 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:07.875 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.906 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:07.934 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.940 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.940 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.966 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:07.970 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.974 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:07.979 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.980 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:07.983 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:07.993 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:07.995 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:08.005 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.008 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.012 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.014 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.017 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.020 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:08.025 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:08.030 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.046 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:08.049 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.060 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.060 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.071 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:08.080 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.093 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.101 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:08.101 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:08.105 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.119 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:08.141 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:08.153 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.153 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.160 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.164 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.174 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.181 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.217 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:08.217 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:08.196 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:08.201 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:08.256 INFO 26087 --- [ocp-schedules-7,1b56e311449c05cb,facba33cd8b78d7f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:10:08.208 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:08.212 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.232 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.255 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.279 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:08.281 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.319 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:08.330 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.357 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:08.368 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:08.372 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.389 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.394 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:08.398 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.401 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.403 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.415 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:08.420 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.438 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.442 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:08.444 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:08.453 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.454 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.478 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.496 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.506 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:08.520 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65 2025-04-08 21:10:08.526 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:08.540 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.561 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.575 WARN 26087 --- [ocp-async-5,0d0c53434dc9e901,da4040529b60f3f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:08.575 INFO 26087 --- [ocp-async-1,735537d3c6fd5c77,96ba48bffd83aa36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:08.567 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:08.572 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:08.595 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,3f80f1a5fbe67029,e46442b5bf35cc80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=3f80f1a5fbe67029, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:08.586 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.617 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,3f80f1a5fbe67029,e46442b5bf35cc80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=3f80f1a5fbe67029, duration=26 ms] 2025-04-08 21:10:08.596 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.608 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.622 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:08.626 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.637 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63 2025-04-08 21:10:08.644 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.644 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.668 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:08.685 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64 2025-04-08 21:10:08.732 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.737 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62 2025-04-08 21:10:08.907 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.922 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:08.933 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:08.957 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:08.966 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:08.996 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.016 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.028 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.037 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:09.041 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:09.050 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.052 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:09.068 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.071 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.081 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.110 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.111 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.118 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.122 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:09.130 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:09.141 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.151 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:09.173 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.191 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.196 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:09.232 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:09.235 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.243 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.251 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.277 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.308 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:09.311 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:09.355 INFO 26087 --- [prometheus-minute-schedule-1,46b7c4675a410947,d2da34310ef046fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:09.318 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:09.322 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:09.367 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.369 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.378 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.381 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.388 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:09.390 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:09.394 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.408 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:09.412 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.417 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.434 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.444 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:09.450 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.452 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:10:09.456 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.489 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.518 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:09.522 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.530 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:09.530 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:09.542 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.545 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.551 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:09.557 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.564 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.571 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.594 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.598 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:10:09.601 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.607 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.609 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:09.618 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:09.620 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.638 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.641 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.642 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.648 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:09.661 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.673 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.675 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:09.678 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65 2025-04-08 21:10:09.682 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:10:09.685 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.725 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:10:09.735 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.749 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.802 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63 2025-04-08 21:10:09.802 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.803 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.812 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.836 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.845 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:09.856 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:09.867 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:09.884 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.894 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:09.922 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,4e32164ddf60b1f3,837b2720a71dbf58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=4e32164ddf60b1f3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:09.939 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,4e32164ddf60b1f3,837b2720a71dbf58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=4e32164ddf60b1f3, duration=21 ms] 2025-04-08 21:10:09.925 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62 2025-04-08 21:10:09.925 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64 2025-04-08 21:10:10.001 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:10:10.002 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:10:10.076 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.079 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.118 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:10.121 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:10.127 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:10:10.131 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:10.158 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.200 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.65 2025-04-08 21:10:10.242 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.246 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.251 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:10.258 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:10.270 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:10:10.272 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:10.291 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.332 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.63 2025-04-08 21:10:10.340 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.345 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.353 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:10.359 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:10.359 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.362 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:10:10.364 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.367 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:10:10.369 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:10:10.382 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:10.416 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:10:10.422 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:10.436 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.442 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.444 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.62 2025-04-08 21:10:10.482 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.64 2025-04-08 21:10:10.558 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.561 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.564 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:10:10.567 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:10:10.575 WARN 26087 --- [ocp-async-2,84fa0730a5fb38d9,05498e32618f444d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:10.575 INFO 26087 --- [ocp-async-1,7ca9289e2b16d488,409b7f1aed2e8f4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:10.570 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:10.604 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:10:10.611 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.652 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.65 2025-04-08 21:10:10.690 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.692 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.696 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 21:10:10.697 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.721 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.737 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,e92493a22d88971d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000046, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:10.742 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:10:10.745 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:10:10.752 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:10.773 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.813 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.63 2025-04-08 21:10:10.815 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.818 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.820 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:10:10.822 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:10:10.847 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.849 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.851 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 21:10:10.852 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.856 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:10.858 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.863 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9a9dd4641112000e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000073, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:10.867 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:10:10.871 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:10:10.877 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:10.881 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.885 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.62 2025-04-08 21:10:10.896 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.936 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.64 2025-04-08 21:10:10.949 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.952 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.966 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 21:10:10.971 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:10.998 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:11.001 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b934ca61576f1fc8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000088, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:11.003 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 21:10:11.012 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000098, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:11.013 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,025da6de9ee68a75] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000080] 2025-04-08 21:10:11.217 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:11.218 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:11.278 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,145e8d702bf250e3,30280b6cfe226f10] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=145e8d702bf250e3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:11.295 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,145e8d702bf250e3,30280b6cfe226f10] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=145e8d702bf250e3, duration=21 ms] 2025-04-08 21:10:11.852 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,a11b3485ec28c203] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000080, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:11.860 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,59b56bc347505d3f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:11.866 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,59b56bc347505d3f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000080, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:11.850+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000080}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:11.875 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,59b56bc347505d3f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000080, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:11.876 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,59b56bc347505d3f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000045, 1000071, 1000087, 1000112] 2025-04-08 21:10:12.355 INFO 26087 --- [prometheus-minute-schedule-1,ac8ee463e18d2b10,2e8616882b3a15f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:12.372 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117740, nodeCount=372, affectRows=372 2025-04-08 21:10:12.375 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117740, nodeCount=371, validCount=371 2025-04-08 21:10:12.576 WARN 26087 --- [ocp-async-9,fb3413a8a96f9162,4c6820d31659a323] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:12.576 INFO 26087 --- [ocp-async-1,ff9f60d3218ec6cd,8d5f734f6c535e90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:12.616 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,dc1684ff8ffbefb6,16480d025fc5c6c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=dc1684ff8ffbefb6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:12.632 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,dc1684ff8ffbefb6,16480d025fc5c6c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=dc1684ff8ffbefb6, duration=19 ms] 2025-04-08 21:10:12.859 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,c065a570c29600bd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000045, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:12.870 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,c065a570c29600bd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000071, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:12.874 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,815f468550652299] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:12.896 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,815f468550652299] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000045, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:12.858+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000045}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:12.898 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,c065a570c29600bd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000087, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:12.898 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,815f468550652299] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=9f485647-1e50-43d2-a243-4dfc7b4c32a3, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000004, authType=OCP-HMACSHA1)) 2025-04-08 21:10:12.900 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,d40cd4f09161f9cb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:12.904 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,c065a570c29600bd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000112, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:12.904 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,d40cd4f09161f9cb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000071, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:12.868+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000071}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:12.905 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,815f468550652299] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=9f485647-1e50-43d2-a243-4dfc7b4c32a3, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000004, authType=OCP-HMACSHA1)) 2025-04-08 21:10:12.906 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,ddb92ea919a8c1d0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:12.912 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,d40cd4f09161f9cb] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=37d72c1a-7205-4f51-805f-18dd8b2c5700, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000002, authType=OCP-HMACSHA1)) 2025-04-08 21:10:12.920 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:12.921 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,815f468550652299] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:1000007 2025-04-08 21:10:12.926 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,ddb92ea919a8c1d0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000087, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:12.897+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000087}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:12.930 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,d40cd4f09161f9cb] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=37d72c1a-7205-4f51-805f-18dd8b2c5700, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000002, authType=OCP-HMACSHA1)) 2025-04-08 21:10:12.943 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000112, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:12.902+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000112}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:12.954 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,ddb92ea919a8c1d0] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=d0d9dc6e-8881-450f-87c1-616e895c4de5, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000005, authType=OCP-HMACSHA1)) 2025-04-08 21:10:12.955 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,815f468550652299] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000045, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:12.971 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,d40cd4f09161f9cb] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:1000008 2025-04-08 21:10:12.975 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=0ac24199-c309-45ed-b205-7da3387d0050, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000003, authType=OCP-HMACSHA1)) 2025-04-08 21:10:13.001 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,ddb92ea919a8c1d0] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=d0d9dc6e-8881-450f-87c1-616e895c4de5, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000005, authType=OCP-HMACSHA1)) 2025-04-08 21:10:13.008 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,d40cd4f09161f9cb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000071, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:13.010 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=0ac24199-c309-45ed-b205-7da3387d0050, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000003, authType=OCP-HMACSHA1)) 2025-04-08 21:10:13.011 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,ddb92ea919a8c1d0] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:1000009 2025-04-08 21:10:13.021 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:1000010 2025-04-08 21:10:13.026 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,ddb92ea919a8c1d0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000087, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:13.028 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000112, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:13.030 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d600db486a5eb145] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000104] 2025-04-08 21:10:13.913 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,5afc42b4e0220678] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000104, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:13.922 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,4498ebc893650ef7] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:13.925 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,4498ebc893650ef7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000104, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:13.911+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000104}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:13.940 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,4dfe844d3878b53f,041f70a3f06494e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=4dfe844d3878b53f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:13.943 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,4498ebc893650ef7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000104, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:10:13.945 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,4498ebc893650ef7] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000061, 1000077, 1000091, 1000115] 2025-04-08 21:10:13.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,4dfe844d3878b53f,041f70a3f06494e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=4dfe844d3878b53f, duration=19 ms] 2025-04-08 21:10:14.218 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:14.219 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:14.576 INFO 26087 --- [ocp-async-1,e4f10c678b8a92aa,6c5b02683c48afe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:14.576 WARN 26087 --- [ocp-async-5,171e1664859f543e,1585473632843e81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:14.922 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,87bffb6ae4900fd7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000061, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:14.929 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,87bffb6ae4900fd7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000077, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:14.931 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:14.935 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,87bffb6ae4900fd7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000091, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:14.935 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000061, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:14.921+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000061}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:14.957 INFO 26087 --- [subtask-coordinator1,8ec1a44aeefd3a62,87bffb6ae4900fd7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000115, operation:EXECUTE, state: RUNNING 2025-04-08 21:10:14.946 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:14.954 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:10:14.969 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000077, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:14.928+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000077}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:14.974 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:14.979 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:10:14.981 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:10:14.983 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000091, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:14.934+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000091}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:14.983 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:10:14.985 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:10:14.986 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:10:14.987 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:10:14.990 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000115, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:10:14.956+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000115}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, executor=192.168.1.61 2025-04-08 21:10:14.993 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:10:14.995 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:10:14.996 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:10:14.998 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:10:15.000 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:10:15.003 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:10:15.006 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:10:15.007 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:10:15.009 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:10:15.011 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:10:15.015 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:10:15.017 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:10:15.026 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:10:15.028 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:10:15.031 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:10:15.048 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:10:15.054 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:10:15.055 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:10:15.056 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:15.069 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:10:15.078 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:15.083 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:10:15.104 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.119 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:10:15.129 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:15.134 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:10:15.137 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.144 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:15.155 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:15.157 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.171 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.178 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:15.181 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.198 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.63 2025-04-08 21:10:15.198 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:15.212 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:15.222 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.225 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.65 2025-04-08 21:10:15.232 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.235 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.64 2025-04-08 21:10:15.236 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.256 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.269 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,c5f073134aa2a05d,93d3a1a74bcf53dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=c5f073134aa2a05d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:15.307 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,c5f073134aa2a05d,93d3a1a74bcf53dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=c5f073134aa2a05d, duration=22 ms] 2025-04-08 21:10:15.277 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.62 2025-04-08 21:10:15.277 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.284 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.355 INFO 26087 --- [prometheus-minute-schedule-1,8e0c65d281ddd47c,688809434275013a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:15.298 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:15.345 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.354 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.364 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.371 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:15.373 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:15.374 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.384 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.392 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.393 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.398 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:15.405 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.63 2025-04-08 21:10:15.413 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.65 2025-04-08 21:10:15.416 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.64 2025-04-08 21:10:15.440 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.472 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.473 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.474 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.481 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.62 2025-04-08 21:10:15.484 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.487 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.492 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:10:15.500 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.503 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:10:15.513 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:10:15.525 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:10:15.525 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:10:15.534 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.540 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.546 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.553 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:10:15.554 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.570 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.63 2025-04-08 21:10:15.572 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.65 2025-04-08 21:10:15.588 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.606 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:10:15.620 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.64 2025-04-08 21:10:15.626 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:10:15.640 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.648 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:10:15.648 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.650 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.659 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.670 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.683 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:10:15.689 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.62 2025-04-08 21:10:15.692 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.694 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:10:15.697 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:10:15.720 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:10:15.732 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.732 ERROR 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:10:15.785 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.799 ERROR 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:10:15.809 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:10:15.812 ERROR 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.825 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.827 ERROR 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.833 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:10:15.855 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.875 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,981ca829eb10d5e4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000061, operation:EXECUTE, state: FAILED 2025-04-08 21:10:15.917 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.920 ERROR 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:10:15.923 ERROR 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.925 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0656dce2f2808f4a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000091, operation:EXECUTE, state: FAILED 2025-04-08 21:10:15.927 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,0d5fe9bf41047d44] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000077, operation:EXECUTE, state: FAILED 2025-04-08 21:10:15.935 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:15.942 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,6d87972544b0ef1c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000115, operation:EXECUTE, state: FAILED 2025-04-08 21:10:16.151 INFO 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=1000007, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000002, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, ssh_port=22, prohibit_rollback=false, service_name=sgkydb:1744016068, ob_connect_address=192.168.1.62:2881, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, createTime=2025-04-08T21:09:56+08:00, startTime=2025-04-08T21:09:56.700+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=1000112, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=34, startTime=2025-04-08T21:10:12.903+08:00, endTime=2025-04-08T21:10:13.024+08:00}, SubtaskInstanceEntity{id=1000059, name=Record root service list, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.RecordRsListTask, seriesId=60, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000045, name=Create random agent credential, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.CreateAgentCredentialTask, seriesId=35, startTime=2025-04-08T21:10:12.858+08:00, endTime=2025-04-08T21:10:12.945+08:00}, SubtaskInstanceEntity{id=1000111, name=Active agent modules, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obagent.ActiveAgentModuleTask, seriesId=39, startTi 2025-04-08 21:10:16.159 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:10:16.159 INFO 26087 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000007, status:FAILED 2025-04-08 21:10:16.512 INFO 26087 --- [ocp-schedules-3,5c377e8d6792e29d,e786a06a2cfd6fb6] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:10:16.512+08:00, tenantSize=0 2025-04-08 21:10:16.517 INFO 26087 --- [ocp-schedules-1,9acade96a77fbbc9,4589b2410da7ebdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:10:16.517 INFO 26087 --- [ocp-schedules-9,0b0cd7314fd7516b,e3472df7e30c4924] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:10:16.521 INFO 26087 --- [ocp-schedules-8,188f0d48f7b69edc,28daf42ed1b4b5d0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:10:16.521 INFO 26087 --- [ocp-schedules-28,79aacea697417f00,74c4a18137939818] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:10:16.521 INFO 26087 --- [ocp-schedules-28,79aacea697417f00,74c4a18137939818] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:10:16.523 INFO 26087 --- [ocp-schedules-1,9acade96a77fbbc9,4589b2410da7ebdb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:10:16.523 INFO 26087 --- [ocp-schedules-19,856ef0521b178ed1,c51cdf5e0ccba433] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:10:16.523 INFO 26087 --- [ocp-schedules-19,856ef0521b178ed1,c51cdf5e0ccba433] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:10:16.524 INFO 26087 --- [ocp-schedules-26,8d9bd4e22a7022ff,00edf6047ce1ae23] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:10:16.524 INFO 26087 --- [ocp-schedules-26,8d9bd4e22a7022ff,00edf6047ce1ae23] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:10:16.524 INFO 26087 --- [ocp-schedules-16,cc9e2afef0b8a876,ef507a28b7eb8564] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:10:16.524 INFO 26087 --- [ocp-schedules-30,52aab8a56cdfbab0,72b44b85a8373417] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:10:16.524 INFO 26087 --- [ocp-schedules-18,904dc779ed3b2133,96de232505e45509] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:10:16.525 INFO 26087 --- [ocp-schedules-30,52aab8a56cdfbab0,72b44b85a8373417] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:16.525 INFO 26087 --- [ocp-schedules-16,cc9e2afef0b8a876,ef507a28b7eb8564] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:16.525 INFO 26087 --- [ocp-schedules-18,904dc779ed3b2133,96de232505e45509] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:10:16.526 INFO 26087 --- [Check status for all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_status_check', labels={app=OB, check_item=cluster connection, failed_reason=-, ob_cluster_id=1744016068, ob_cluster_name=sgkydb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744117816 : 2025-04-08 13:10:16 2025-04-08 21:10:16.526 INFO 26087 --- [ocp-schedules-9,0b0cd7314fd7516b,e3472df7e30c4924] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:10:16.527 INFO 26087 --- [ocp-schedules-29,029d393679e60e8e,8aa032e4d7c97801] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:10:16.531 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:10:16.531 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:10:16.532 INFO 26087 --- [Sync all clusters1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='ob_cluster_sync', labels={app=OB, failed_reason=-, ob_cluster_id=1744016068, ob_cluster_name=sgkydb, ob_cluster_status=TAKINGOVER}}, value: -1, timestamp: 1744117816 : 2025-04-08 13:10:16 2025-04-08 21:10:16.544 INFO 26087 --- [ocp-async-4,dcf856a5165032b6,0ed83df7090af39c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:10:16.556 INFO 26087 --- [ocp-schedules-17,ee557a9ea13bde84,577226a9921ba610] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:10:16.557 INFO 26087 --- [ocp-schedules-17,ee557a9ea13bde84,577226a9921ba610] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:10:16.576 INFO 26087 --- [ocp-async-7,3f4b8056713c9543,aaa7856e9b557eff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:16.576 WARN 26087 --- [ocp-async-9,0c7ae141d0d9f781,57104b468867ff9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:16.579 INFO 26087 --- [ocp-schedules-3,2357375077eafc50,1260fc7af3d9d965] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:10:16.587 INFO 26087 --- [ocp-schedules-24,759fdeb0e93b904b,515a2bacea027e26] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:10:16.596 INFO 26087 --- [ocp-schedules-19,522d635299d1ca26,148898db46f7c022] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:10:16.596+08:00, clusters=[1000002] 2025-04-08 21:10:16.596 INFO 26087 --- [ocp-schedules-28,d40c77842ab047ff,9288e5cd8d73288f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:10:16.596+08:00, clusters=[1000002] 2025-04-08 21:10:16.597 INFO 26087 --- [ocp-schedules-28,d40c77842ab047ff,9288e5cd8d73288f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:10:16.597 INFO 26087 --- [ocp-schedules-8,370efb76c6f05868,27a0d75f92beabcb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:10:16.598 INFO 26087 --- [ocp-schedules-19,522d635299d1ca26,148898db46f7c022] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:10:16.602 INFO 26087 --- [ocp-schedules-9,f3b2969b56e5ee37,40cc8ab4aee620e4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:10:16.602 INFO 26087 --- [ocp-schedules-9,f3b2969b56e5ee37,40cc8ab4aee620e4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:10:16.605 INFO 26087 --- [ocp-schedules-29,9bca11f1fd481c81,ea997934eb6f1acd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:10:16.619 INFO 26087 --- [ocp-schedules-1,d6b9704cd96f2aa4,d751c5bb7775783d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:10:16.619 INFO 26087 --- [ocp-schedules-1,d6b9704cd96f2aa4,d751c5bb7775783d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:10:16.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:10:16.624534+08:00, lastProcessTime=2025-04-08T21:10:06.624311+08:00 2025-04-08 21:10:16.626 INFO 26087 --- [ocp-schedules-1,d6b9704cd96f2aa4,d751c5bb7775783d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:10:16.626 INFO 26087 --- [ocp-schedules-1,d6b9704cd96f2aa4,d751c5bb7775783d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:10:16.626 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,ef95140d7cc5ca92,a5eb9afe4b5074aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=ef95140d7cc5ca92, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:10:16.626 INFO 26087 --- [ocp-schedules-27,eda334e9e559b223,f80287526b5a8814] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:10:16.626 INFO 26087 --- [ocp-schedules-27,eda334e9e559b223,f80287526b5a8814] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:10:16.631 INFO 26087 --- [ocp-schedules-5,030c5c677e12e798,e6255534ad67256e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:10:16.632 INFO 26087 --- [ocp-schedules-5,030c5c677e12e798,e6255534ad67256e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:16.642 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,ef95140d7cc5ca92,a5eb9afe4b5074aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=ef95140d7cc5ca92, duration=20 ms] 2025-04-08 21:10:16.644 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=sgkydb, obClusterId=1744016068, tenantName=sys, username=root, address=192.168.1.63, port=2881, database=oceanbase) 2025-04-08 21:10:16.644 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=sgkydb, obClusterId=1744016068, tenantName=sys, username=root, address=192.168.1.65, port=2881, database=oceanbase) 2025-04-08 21:10:16.644 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=sgkydb, obClusterId=1744016068, tenantName=sys, username=root, address=192.168.1.62, port=2881, database=oceanbase) 2025-04-08 21:10:16.644 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=sgkydb, obClusterId=1744016068, tenantName=sys, username=root, address=192.168.1.64, port=2881, database=oceanbase) 2025-04-08 21:10:16.644 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=192.168.1.63, port=2881), ObServerAddr(address=192.168.1.65, port=2881), ObServerAddr(address=192.168.1.62, port=2881), ObServerAddr(address=192.168.1.64, port=2881)], username=root, tenantName=sys, clusterName=sgkydb, obClusterId=1744016068, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:10:16.651 INFO 26087 --- [obsdk-init-thread-6,7af8f2cd058c093b,80b29765af3449fe] com.alibaba.druid.pool.DruidDataSource : {dataSource-8,ob-connector-datasource_direct_sys_root_192.168.1.63_2881} inited 2025-04-08 21:10:16.652 INFO 26087 --- [obsdk-init-thread-6,7af8f2cd058c093b,80b29765af3449fe] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=192.168.1.63, port=2881, obsAddrList=[ObServerAddr(address=192.168.1.63, port=2881), ObServerAddr(address=192.168.1.65, port=2881), ObServerAddr(address=192.168.1.62, port=2881), ObServerAddr(address=192.168.1.64, port=2881)], username=root, tenantName=sys, clusterName=sgkydb, obClusterId=1744016068, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-04-08 21:10:16.652 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-04-08 21:10:16.654 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:16.654 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:10:16.656 INFO 26087 --- [ocp-schedules-7,d080e8aacf9af905,ea068c8640cd6e6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:10:16.657 INFO 26087 --- [ocp-schedules-13,990d08345776e4e8,921416693fdc823e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:10:16.658 INFO 26087 --- [ocp-schedules-7,d080e8aacf9af905,ea068c8640cd6e6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:10:16.658 INFO 26087 --- [ocp-schedules-7,d080e8aacf9af905,ea068c8640cd6e6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:10:16.669 INFO 26087 --- [ocp-schedules-23,6c9c675b7f9c2598,e0567d3e83355d51] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:10:16.669 INFO 26087 --- [ocp-schedules-23,6c9c675b7f9c2598,e0567d3e83355d51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:16.684 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:10:16.684 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:10:16.693 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:10:16.694 ERROR 26087 --- [ocp-schedules-23,6c9c675b7f9c2598,e0567d3e83355d51] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:10:16.695 INFO 26087 --- [ocp-schedules-23,6c9c675b7f9c2598,e0567d3e83355d51] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:10:16.727 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:10:16.624534+08:00 2025-04-08 21:10:16.745 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.o.m.p.OcpMetricPublishServiceImpl : Register meter successfully, metric = Metric{name='cluster_cgroup_value', labels={app=OB, ob_cluster_id=1744016068, ob_cluster_name=sgkydb}}, value: 0, timestamp: 1744117816 : 2025-04-08 13:10:16 2025-04-08 21:10:16.805 INFO 26087 --- [ocp-schedules-25,5cd3d053c6ece813,290547cf70b398a7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:10:16.805 INFO 26087 --- [ocp-schedules-25,5cd3d053c6ece813,290547cf70b398a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:10:16.808 INFO 26087 --- [ocp-schedules-25,5cd3d053c6ece813,290547cf70b398a7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:10:16.808 INFO 26087 --- [ocp-schedules-25,5cd3d053c6ece813,290547cf70b398a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:10:16.811 INFO 26087 --- [ocp-schedules-25,5cd3d053c6ece813,290547cf70b398a7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:10:16.811 INFO 26087 --- [ocp-schedules-25,5cd3d053c6ece813,290547cf70b398a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:10:16.908 INFO 26087 --- [ocp-schedules-11,fb081b255a58e40b,c8be674681e94119] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:10:16.919 INFO 26087 --- [ocp-schedules-11,fb081b255a58e40b,c8be674681e94119] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:10:16.919 INFO 26087 --- [ocp-schedules-11,fb081b255a58e40b,c8be674681e94119] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:10:17.007 INFO 26087 --- [ocp-schedules-12,175e85c5b4088251,6636b7d703339fe4] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61, current targets ,,,,,C_sgkydb-1744016068-PRIMARY,C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,C_sgkydb-1744016068/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.147 INFO 26087 --- [ocp-schedules-24,4999882a2a1bd4e4,644d68603cf6b633] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:10:17.148 INFO 26087 --- [ocp-schedules-24,4999882a2a1bd4e4,644d68603cf6b633] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:10:17.192 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:10:17.193 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:10:17.194 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:10:17.195 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:10:17.196 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:10:17.197 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:10:17.198 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:10:17.199 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:10:17.200 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:10:17.201 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:10:17.202 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:10:17.203 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:10:17.204 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:10:17.205 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:10:17.206 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:10:17.207 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:10:17.208 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:10:17.209 INFO 26087 --- [ocp-schedules-14,86b2b7ceb32b6b9e,e3dc0a64161e2a01] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:10:17.219 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:17.219 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:17.370 INFO 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:10:17.370 INFO 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:10:17.370 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:10:17.370 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:10:17.370 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:10:17.373 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:10:17.373 INFO 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:10:17.373 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:10:17.374 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:10:17.374 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:10:17.374 WARN 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:10:17.374 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:10:17.375 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:10:17.375 INFO 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:10:17.375 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:10:17.375 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:10:17.375 WARN 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:10:17.376 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:10:17.376 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:10:17.377 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:10:17.377 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:10:17.385 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.63, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:8 2025-04-08 21:10:17.385 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:10:17.386 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.64, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:9 2025-04-08 21:10:17.386 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:10:17.387 INFO 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.62, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:10 2025-04-08 21:10:17.387 INFO 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:10:17.389 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.65, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:11 2025-04-08 21:10:17.389 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:10:17.444 WARN 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:10:17.446 ERROR 26087 --- [ocp-schedules-19,3104e0437c7dfdf7,a84a591c9dde8f5c] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2025-04-08 21:10:18.280 INFO 26087 --- [ocp-schedules-7,05452e5e401e0fbf,1fb43f0cb4a5c55d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:10:18.355 INFO 26087 --- [prometheus-minute-schedule-1,396d73b4b3ae84f0,90460c50ab70ccb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:18.576 WARN 26087 --- [ocp-async-10,0df264c69acd75bf,10c7be3dde401255] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:18.576 INFO 26087 --- [ocp-async-5,0331398e3dc5f63b,4239384931271c09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:20.220 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:20.220 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:20.576 WARN 26087 --- [ocp-async-5,ba78aff80138bd09,7c21f05a8b0d0f6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:20.576 INFO 26087 --- [ocp-async-1,3b71f89c3cd70eb2,15384a1a4dd27f0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:10:21.355 INFO 26087 --- [prometheus-minute-schedule-1,6e92d2a6e7210f9c,33c4aa2d310a7909] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:22.576 WARN 26087 --- [ocp-async-1,91e4507eff16f10a,a98dad55553a2c11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:22.576 INFO 26087 --- [ocp-async-6,07595535d0f0375e,aacdf2069dccdf01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:23.220 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:23.221 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:24.355 INFO 26087 --- [prometheus-minute-schedule-1,8dd41d80db135f81,410732f4bb73aa2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:24.577 WARN 26087 --- [ocp-async-3,8b0eff2c12ef890d,daf3beecf9de9a90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:24.577 INFO 26087 --- [ocp-async-8,4a17e4177732721e,604485fd2b5c4252] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:26.221 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:26.222 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:26.526 INFO 26087 --- [ocp-schedules-18,904dc779ed3b2133,96de232505e45509] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:10:26.526 INFO 26087 --- [ocp-schedules-30,52aab8a56cdfbab0,72b44b85a8373417] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:10:26.544 INFO 26087 --- [ocp-async-5,8b61ea53dd80ae4c,ad489b3a3cfc2209] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:10:26.577 WARN 26087 --- [ocp-async-1,7d26a85921694a00,81b66daf876edfe6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:26.577 INFO 26087 --- [ocp-async-2,7da64839c1271cce,24172ff334b8aef7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:26.606 INFO 26087 --- [ocp-schedules-15,bd2ccd8a5df1731f,ba8303b9d34619a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:10:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:10:26.623429+08:00, lastProcessTime=2025-04-08T21:10:16.624534+08:00 2025-04-08 21:10:26.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:10:26.623429+08:00 2025-04-08 21:10:27.154 INFO 26087 --- [ocp-schedules-26,fd18a2125f202582,7235570afed35efc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:10:27.155 INFO 26087 --- [ocp-schedules-26,fd18a2125f202582,7235570afed35efc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:10:27.355 INFO 26087 --- [prometheus-minute-schedule-1,bdbac3fc979446ed,6a24bbb0c1abb6c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:28.342 INFO 26087 --- [ocp-schedules-28,128f0a7893eabc52,a4e1c34f8bfa4792] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:10:28.577 WARN 26087 --- [ocp-async-2,51ad9cc2c1f63736,12d4ac2602580266] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:28.577 INFO 26087 --- [ocp-async-6,c50f232659bcb0fb,028a1c343ab6bb76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:29.222 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:29.222 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:30.355 INFO 26087 --- [prometheus-minute-schedule-1,762c77199f2e2f38,e7a9142c9c7e617f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:30.577 WARN 26087 --- [ocp-async-7,16c3ae592e1463a4,f40649b883005eaf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:30.577 INFO 26087 --- [ocp-async-6,f34cb99ebf176905,9cdce897ace600ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:32.222 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:32.223 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:32.577 WARN 26087 --- [ocp-async-6,b8d0242ae12630d4,2daf762526bcfd5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:32.577 INFO 26087 --- [ocp-async-5,9ee48845f90e2558,de7f948e862061a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:33.355 INFO 26087 --- [prometheus-minute-schedule-1,e101f4f9ca614d1d,520a021678f09b70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:34.577 WARN 26087 --- [ocp-async-2,40f49e8463a45bff,dc0c83016c3d8282] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:34.577 INFO 26087 --- [ocp-async-5,d0c014a50c26930a,bd8905c7c012af78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:35.223 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:35.224 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:36.355 INFO 26087 --- [prometheus-minute-schedule-1,6edcd7f845e95c57,182f1b2dcd624418] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:36.544 INFO 26087 --- [ocp-async-10,ae27d662a87b6cd1,efc3da302007d5de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:10:36.578 WARN 26087 --- [ocp-async-7,33b6bd9cb8332626,165ceb050c84ff78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:36.578 INFO 26087 --- [ocp-async-8,d9dcfc4d5c5aa984,856de31e004aaa30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:36.607 INFO 26087 --- [ocp-schedules-20,6bbcd8010bb5991e,9eaf481a3b8a7d46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:10:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:10:36.622456+08:00, lastProcessTime=2025-04-08T21:10:26.623429+08:00 2025-04-08 21:10:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:10:36.622456+08:00 2025-04-08 21:10:36.921 INFO 26087 --- [ocp-schedules-25,62978bb7b8665e8b,11328f785fe1cb97] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:10:36.933 INFO 26087 --- [ocp-schedules-25,62978bb7b8665e8b,11328f785fe1cb97] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:10:36.933 INFO 26087 --- [ocp-schedules-25,62978bb7b8665e8b,11328f785fe1cb97] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:10:37.170 INFO 26087 --- [ocp-schedules-22,00570082d6fbc05a,6d2ec030c25e1ef6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:10:37.171 INFO 26087 --- [ocp-schedules-22,00570082d6fbc05a,6d2ec030c25e1ef6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:10:38.224 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:38.224 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:38.367 INFO 26087 --- [ocp-schedules-25,33195edd6ed7b555,acefafb3ca809615] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:10:38.578 INFO 26087 --- [ocp-async-6,61da01f5077173bb,48a6f917e03177e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:38.578 WARN 26087 --- [ocp-async-9,cbf4234154703849,e572cd226730a5ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:39.355 INFO 26087 --- [prometheus-minute-schedule-1,e0577c17dc56c9a7,97f0710d7647721e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:40.001 INFO 26087 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:10:40.001 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:10:40.578 WARN 26087 --- [ocp-async-5,ed51d533f4be8efa,7bd8733043d68321] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:40.578 INFO 26087 --- [ocp-async-2,285fcef224e00f3d,98604cfa3ef03460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:41.225 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:41.225 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:42.355 INFO 26087 --- [prometheus-minute-schedule-1,f0d080c0d0dd1159,4e2781297816406e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:42.578 WARN 26087 --- [ocp-async-7,39d3b9f6f2e67837,693fa04bed704c9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:42.578 INFO 26087 --- [ocp-async-1,888c688efe800a9a,5fe31bca4b8ccf0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:44.225 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:44.226 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:44.578 WARN 26087 --- [ocp-async-8,19317b0af56ab7af,ff9b6a13796b280d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:44.578 INFO 26087 --- [ocp-async-4,b4a083aed28f8b97,8321b8d98b7ea39b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:45.355 INFO 26087 --- [prometheus-minute-schedule-1,60d0b85797796d7a,8e29e94da223f1dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:46.521 INFO 26087 --- [ocp-schedules-1,e9191cc445dbf481,e7f5562a08679181] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:10:46.521 INFO 26087 --- [ocp-schedules-1,e9191cc445dbf481,e7f5562a08679181] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:10:46.522 INFO 26087 --- [ocp-schedules-12,78383d437daf5138,9338f85cec4b1cca] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:10:46.524 INFO 26087 --- [ocp-schedules-13,26a7df5327c41ca2,f65abd948c1a75ff] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:10:46.524 INFO 26087 --- [ocp-schedules-2,1b2bfd77a5a402a0,98414a9b1371908a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:10:46.524 INFO 26087 --- [ocp-schedules-2,1b2bfd77a5a402a0,98414a9b1371908a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:10:46.524 INFO 26087 --- [ocp-schedules-13,26a7df5327c41ca2,f65abd948c1a75ff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:10:46.525 INFO 26087 --- [ocp-schedules-8,b0a27d2e5c6220cd,a83f7bce1cdf09c6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:10:46.525 INFO 26087 --- [ocp-schedules-28,e289573e338f7802,3a13d77a90b82ed8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:10:46.526 INFO 26087 --- [ocp-schedules-27,49a9395fda778571,4b8bfabdf3ba41ad] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:10:46.526 INFO 26087 --- [ocp-schedules-8,b0a27d2e5c6220cd,a83f7bce1cdf09c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:46.526 INFO 26087 --- [ocp-schedules-28,e289573e338f7802,3a13d77a90b82ed8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:46.526 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:10:46.532 INFO 26087 --- [ocp-schedules-16,cc9e2afef0b8a876,ef507a28b7eb8564] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:10:46.533 INFO 26087 --- [ocp-schedules-16,0c79d0d01603f829,6219ba3556bb22f5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:10:46.534 INFO 26087 --- [ocp-schedules-16,0c79d0d01603f829,6219ba3556bb22f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:46.541 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:10:46.541 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:10:46.544 INFO 26087 --- [ocp-async-4,007594e084bfdbe7,cfc155bb42e985cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:10:46.579 WARN 26087 --- [ocp-async-2,e657a443ff51d0ca,5a05022f429b2891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:46.579 INFO 26087 --- [ocp-async-3,947c0bf8c7f66cbe,6e631efbe7a1088e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:46.589 INFO 26087 --- [ocp-schedules-14,def5e7e748fbe220,33949f5d094e5416] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:10:46.609 INFO 26087 --- [ocp-schedules-14,9d23ff1bdf4c559a,6b66ab7df20eb2f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:10:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:10:46.623108+08:00, lastProcessTime=2025-04-08T21:10:36.622456+08:00 2025-04-08 21:10:46.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:10:46.623108+08:00 2025-04-08 21:10:47.176 INFO 26087 --- [ocp-schedules-24,db2d51dd94a4496e,7a7452bf8f9d4481] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:10:47.177 INFO 26087 --- [ocp-schedules-24,db2d51dd94a4496e,7a7452bf8f9d4481] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:10:47.226 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:47.226 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:48.355 INFO 26087 --- [prometheus-minute-schedule-1,083c46de0bd7c94e,269efefbd601a435] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:48.396 INFO 26087 --- [ocp-schedules-12,d5c23ca942456b04,cf8f4c79acceabc6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:10:48.579 WARN 26087 --- [ocp-async-7,ea3bc066b31427c1,8dbaf99e7df0b703] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:48.579 INFO 26087 --- [ocp-async-10,2ba7ff1411c260f7,6ce0be1808261d10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:50.227 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:50.227 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:50.579 WARN 26087 --- [ocp-async-8,a6fb77542466d1c7,c2fd030a0f10d90f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:50.579 INFO 26087 --- [ocp-async-9,07da9cec7f62edbe,3786ebd45c371c36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:51.355 INFO 26087 --- [prometheus-minute-schedule-1,0abd566b1b11aee8,be3d7b48e3e89bc2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:52.579 WARN 26087 --- [ocp-async-5,e35da2264a3a9fd9,fb000717a2885dd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:52.579 INFO 26087 --- [ocp-async-2,e0c87ef5983d7465,4e7ca60d93aa38d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:53.227 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:53.228 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:54.355 INFO 26087 --- [prometheus-minute-schedule-1,3953159e4240dcec,453fe88e848e4f73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:54.579 WARN 26087 --- [ocp-async-7,0dde0521df300a27,f635ac5e6addb9db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:54.579 INFO 26087 --- [ocp-async-3,b246957e18352219,c2cecd854f99645f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:56.228 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:56.228 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:10:56.527 INFO 26087 --- [ocp-schedules-8,b0a27d2e5c6220cd,a83f7bce1cdf09c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:10:56.527 INFO 26087 --- [ocp-schedules-28,e289573e338f7802,3a13d77a90b82ed8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:10:56.544 INFO 26087 --- [ocp-async-3,c06e74fc30d684f0,2c70d720084fa5bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:10:56.560 INFO 26087 --- [ocp-schedules-12,95e48a0ea724cb2e,5f71d68dd80075c1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:10:56.560 INFO 26087 --- [ocp-schedules-12,95e48a0ea724cb2e,5f71d68dd80075c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:10:56.561 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:10:56.579 WARN 26087 --- [ocp-async-6,6e7dce97f5323dfd,816c5bbe6bf9b6f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:56.579 INFO 26087 --- [ocp-async-8,fbcedd1816029272,8032723688037820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:56.610 INFO 26087 --- [ocp-schedules-23,2d763cc2bc3038bc,825fdb55a1388ef1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:10:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:10:56.622415+08:00, lastProcessTime=2025-04-08T21:10:46.623108+08:00 2025-04-08 21:10:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:10:56.622415+08:00 2025-04-08 21:10:56.935 INFO 26087 --- [ocp-schedules-4,ac58cc648499c1bd,7a099f7ceeacb032] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:10:56.948 INFO 26087 --- [ocp-schedules-4,ac58cc648499c1bd,7a099f7ceeacb032] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:10:56.948 INFO 26087 --- [ocp-schedules-4,ac58cc648499c1bd,7a099f7ceeacb032] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:10:57.182 INFO 26087 --- [ocp-schedules-24,ded64d2e2677b5b1,27f32101badc1bda] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:10:57.184 INFO 26087 --- [ocp-schedules-24,ded64d2e2677b5b1,27f32101badc1bda] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:10:57.355 INFO 26087 --- [prometheus-minute-schedule-1,f995649bd5414956,a6ff60016339dd98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117860, interval=60 2025-04-08 21:10:58.419 INFO 26087 --- [ocp-schedules-4,3b97120357d10292,e7b62cc44cdb2721] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:10:58.580 WARN 26087 --- [ocp-async-5,9acb4617a458f873,e1ce70ec5cb36f4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:10:58.580 INFO 26087 --- [ocp-async-4,7a759197e198eb9d,0bbc39c12b596ce5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:10:59.229 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:10:59.229 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:00.355 INFO 26087 --- [prometheus-minute-schedule-1,2608e694d672cd50,44f7f7917410cbd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:00.580 WARN 26087 --- [ocp-async-4,d7e357fc44231dab,faf911e17e0b7894] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:00.580 INFO 26087 --- [ocp-async-1,7ed816c56d524b99,5ba4c22192c8e346] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:02.229 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:02.230 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:02.580 WARN 26087 --- [ocp-async-6,1e46dcad506a6de7,de56293a414d5978] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:02.580 INFO 26087 --- [ocp-async-3,379529a8a0e1524f,83d67faafdae4f84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:03.355 INFO 26087 --- [prometheus-minute-schedule-1,1a7af5abe0845748,4faf7eeeabbe70e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:04.580 INFO 26087 --- [ocp-async-3,8faa2913bc06b4bb,23790e99cef5c076] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:04.580 WARN 26087 --- [ocp-async-5,a442d4273e5d6466,3ce41d82ce8d9b74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:05.230 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:05.230 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:06.355 INFO 26087 --- [prometheus-minute-schedule-1,0e9b9cf61c188d10,b1b28a3d68aa16af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:06.355 INFO 26087 --- [prometheus-cache-monitor1,199c0147b4a1e282,7e54b8da3cfc8ddc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1486,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:11:06.355 INFO 26087 --- [prometheus-cache-monitor1,199c0147b4a1e282,7e54b8da3cfc8ddc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:11:06.545 INFO 26087 --- [ocp-async-3,bca2f4240130a8a2,469a87602adea936] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:11:06.561 INFO 26087 --- [ocp-schedules-12,95e48a0ea724cb2e,5f71d68dd80075c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:11:06.580 WARN 26087 --- [ocp-async-10,9fc185900b33e0dd,db231cdefae813b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:06.580 INFO 26087 --- [ocp-async-4,8f848e3b5f1324e1,c9275cd4ba7c7401] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:06.611 INFO 26087 --- [ocp-schedules-27,6c257f84e84fa085,a2b8b3c90aca0f5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:11:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:11:06.622230+08:00, lastProcessTime=2025-04-08T21:10:56.622415+08:00 2025-04-08 21:11:06.758 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:11:06.622230+08:00 2025-04-08 21:11:07.208 INFO 26087 --- [ocp-schedules-13,091af1d7858fed2d,0dd6fa323ad215c9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:11:07.210 INFO 26087 --- [ocp-schedules-13,091af1d7858fed2d,0dd6fa323ad215c9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:11:08.231 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:08.231 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:08.447 INFO 26087 --- [ocp-schedules-1,5ef1e8cfc7637918,237099ec29c8612a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:11:08.580 WARN 26087 --- [ocp-async-1,48bdf58832f055dc,21726c088a56cca9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:08.580 INFO 26087 --- [ocp-async-9,b4e6c06c76ac91a5,fbbbab4f6078c1b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:09.355 INFO 26087 --- [prometheus-minute-schedule-1,a3515664dd14abe6,2ded43949acd79aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:10.002 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:11:10.002 INFO 26087 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:11:10.580 WARN 26087 --- [ocp-async-2,0eecd96be9a95636,cadf16c6e59d2181] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:10.581 INFO 26087 --- [ocp-async-8,bee4d31cfba0c674,d3511e4fef15aadf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:11:11.231 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:11.232 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:12.355 INFO 26087 --- [prometheus-minute-schedule-1,9a093eb24c588c91,e2c397f73cd037ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=24, affectRows=24 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=25, affectRows=25 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=34, affectRows=34 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=58, affectRows=58 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=24, validCount=24 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=41, affectRows=41 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=25, validCount=25 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=57, affectRows=57 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=50, affectRows=50 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=34, validCount=34 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=53, affectRows=53 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=58, validCount=58 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=41, validCount=41 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=50, validCount=50 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117800, nodeCount=32, affectRows=32 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=56, validCount=56 2025-04-08 21:11:12.369 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=52, validCount=52 2025-04-08 21:11:12.370 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117800, nodeCount=31, validCount=31 2025-04-08 21:11:12.581 WARN 26087 --- [ocp-async-7,1814560dfdb8517b,21a1cd39d6383b56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:12.581 INFO 26087 --- [ocp-async-6,374332bb85be2b77,1d6287ef49fb6bd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:14.232 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:14.232 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:14.581 WARN 26087 --- [ocp-async-5,df1ef335c8ff032f,eeb47f613ad8e7f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:14.581 INFO 26087 --- [ocp-async-1,8f40bca1c37e6bb0,88d9ab1cb501ef7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:15.355 INFO 26087 --- [prometheus-minute-schedule-1,15b2d687c0e9380a,5982d0de5f0588c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:16.517 INFO 26087 --- [ocp-schedules-8,3e1aacdbe90ce309,a0fa8b72dff2c304] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:11:16.518 INFO 26087 --- [ocp-schedules-29,718beb4d37b5de39,1bb524e5d063d355] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:11:16.521 INFO 26087 --- [ocp-schedules-6,ec481570fbe21c45,4b5e2fc4126b34e6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:11:16.521 INFO 26087 --- [ocp-schedules-17,0c80e5e8d6d1e762,6d3c26782da062ed] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:11:16.521 INFO 26087 --- [ocp-schedules-17,0c80e5e8d6d1e762,6d3c26782da062ed] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:11:16.524 INFO 26087 --- [ocp-schedules-9,74e7cb75d4737c69,4e72ca224bb71324] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:11:16.524 INFO 26087 --- [ocp-schedules-9,74e7cb75d4737c69,4e72ca224bb71324] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:11:16.524 INFO 26087 --- [ocp-schedules-18,25f44d629c80a957,b65ec4db7af51b16] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:11:16.524 INFO 26087 --- [ocp-schedules-21,4999daa259233cf0,3302aa3f81dc0041] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:11:16.525 INFO 26087 --- [ocp-schedules-18,25f44d629c80a957,b65ec4db7af51b16] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:16.525 INFO 26087 --- [ocp-schedules-21,4999daa259233cf0,3302aa3f81dc0041] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:16.525 INFO 26087 --- [ocp-schedules-29,718beb4d37b5de39,1bb524e5d063d355] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:11:16.525 INFO 26087 --- [ocp-schedules-11,3421076fe04fdac4,c604cdfcd49bb7e6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:11:16.525 INFO 26087 --- [ocp-schedules-11,3421076fe04fdac4,c604cdfcd49bb7e6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:11:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:11:16.541 INFO 26087 --- [ocp-schedules-16,0c79d0d01603f829,6219ba3556bb22f5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:11:16.542 INFO 26087 --- [ocp-schedules-16,be1c29b362e586fd,74d29eb0d980e258] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:11:16.542 INFO 26087 --- [ocp-schedules-16,be1c29b362e586fd,74d29eb0d980e258] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:16.545 INFO 26087 --- [ocp-async-10,298d9fa6b29dc96e,c7e34901e62ac16b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:11:16.549 INFO 26087 --- [ocp-schedules-8,3e1aacdbe90ce309,a0fa8b72dff2c304] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:11:16.549 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:11:16.549 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:11:16.561 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:11:16.561 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:11:16.561 INFO 26087 --- [ocp-schedules-25,aa0dd17e3fbd7581,68d7bc8b2535285a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:11:16.566 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:11:16.566 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:11:16.566 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:11:16.568 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:11:16.568 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:11:16.568 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:11:16.570 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:11:16.570 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:11:16.570 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:11:16.571 INFO 26087 --- [ocp-schedules-10,f0a119136ef276e2,8e749d5ce71a5fae] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:11:16.580 INFO 26087 --- [ocp-schedules-2,a4909c0fe023c4a7,34dc24a6fdf5e6c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:11:16.581 INFO 26087 --- [ocp-async-7,dd322efa33d0e22c,1cfd75cba7d65543] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:16.581 WARN 26087 --- [ocp-async-3,ab273bbf6f411627,e7e563909e61e0c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:16.584 INFO 26087 --- [ocp-schedules-17,1fec98fc60dcc964,9836f238614ffcf3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1494,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":372,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:11:16.584 INFO 26087 --- [ocp-schedules-17,1fec98fc60dcc964,9836f238614ffcf3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:11:16.590 INFO 26087 --- [ocp-schedules-24,694fab3a7b2d9e9a,c9d1aad97994cfa7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:11:16.603 INFO 26087 --- [ocp-schedules-29,9a793ec82a2b8dd4,29fd2618735fb167] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:11:16.603+08:00, clusters=[1000002] 2025-04-08 21:11:16.603 INFO 26087 --- [ocp-schedules-29,9a793ec82a2b8dd4,29fd2618735fb167] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:11:16.603 INFO 26087 --- [ocp-schedules-4,281f6de7522be436,a62700d202e9916b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:11:16.603+08:00, clusters=[1000002] 2025-04-08 21:11:16.603 INFO 26087 --- [ocp-schedules-11,02c826ef6816d2dc,51732b8797a12f4d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:11:16.604 INFO 26087 --- [ocp-schedules-23,d5560a4eda754820,6b63e9cbf9839e0b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:11:16.604 INFO 26087 --- [ocp-schedules-23,d5560a4eda754820,6b63e9cbf9839e0b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:11:16.605 INFO 26087 --- [ocp-schedules-4,281f6de7522be436,a62700d202e9916b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:11:16.612 INFO 26087 --- [ocp-schedules-20,84cdf28af3101aed,81768f81e7853cb4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:11:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:11:16.622978+08:00, lastProcessTime=2025-04-08T21:11:06.622230+08:00 2025-04-08 21:11:16.626 INFO 26087 --- [ocp-schedules-15,ee060823e68e9b08,50dbf4136e3bc214] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:11:16.626 INFO 26087 --- [ocp-schedules-15,ee060823e68e9b08,50dbf4136e3bc214] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:11:16.628 INFO 26087 --- [ocp-schedules-30,1ee2985a30395a71,ff79d1a797326e41] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:11:16.628 INFO 26087 --- [ocp-schedules-30,1ee2985a30395a71,ff79d1a797326e41] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:11:16.632 INFO 26087 --- [ocp-schedules-15,ee060823e68e9b08,50dbf4136e3bc214] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:11:16.632 INFO 26087 --- [ocp-schedules-15,ee060823e68e9b08,50dbf4136e3bc214] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:11:16.658 INFO 26087 --- [ocp-schedules-7,f64e7eec5adc2831,159d172aaa12c210] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:11:16.659 INFO 26087 --- [ocp-schedules-7,f64e7eec5adc2831,159d172aaa12c210] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:11:16.659 INFO 26087 --- [ocp-schedules-7,f64e7eec5adc2831,159d172aaa12c210] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:11:16.661 INFO 26087 --- [ocp-schedules-22,32a4a5182668b53a,529793f54b95ce1d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:11:16.704 INFO 26087 --- [ocp-schedules-27,0e0a38b5e42d7d33,40961f17d875c462] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:11:16.705 INFO 26087 --- [ocp-schedules-27,0e0a38b5e42d7d33,40961f17d875c462] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:16.725 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:11:16.726 ERROR 26087 --- [ocp-schedules-27,0e0a38b5e42d7d33,40961f17d875c462] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:11:16.726 INFO 26087 --- [ocp-schedules-27,0e0a38b5e42d7d33,40961f17d875c462] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:11:16.745 INFO 26087 --- [ocp-schedules-5,030c5c677e12e798,e6255534ad67256e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:11:16.747 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:11:16.622978+08:00 2025-04-08 21:11:16.781 INFO 26087 --- [ocp-schedules-1,31073a7a54fd787e,84497aac3e91820c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:11:16.782 INFO 26087 --- [ocp-schedules-1,31073a7a54fd787e,84497aac3e91820c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:16.797 ERROR 26087 --- [ocp-schedules-1,31073a7a54fd787e,84497aac3e91820c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:11:16.797 INFO 26087 --- [ocp-schedules-1,31073a7a54fd787e,84497aac3e91820c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:11:16.813 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:11:16.813 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:11:16.813 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:11:16.813 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766200877631795200 2025-04-08 21:11:16.814 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=3, maxSyncedRecordId=766208338840276992 2025-04-08 21:11:16.814 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:11:16.814 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766198483195916288 2025-04-08 21:11:16.817 INFO 26087 --- [ocp-schedules-25,40971595f4b3f8e9,61cff40c09b422c0] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=130, maxSyncedRecordId=766208550866538496 2025-04-08 21:11:16.949 INFO 26087 --- [ocp-schedules-10,58d71f189c956c3c,9e878218e8df44a6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:11:16.960 INFO 26087 --- [ocp-schedules-10,58d71f189c956c3c,9e878218e8df44a6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:11:16.960 INFO 26087 --- [ocp-schedules-10,58d71f189c956c3c,9e878218e8df44a6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:11:17.215 INFO 26087 --- [ocp-schedules-9,c9a2d950e080e247,ffc27705a4070bca] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:11:17.216 INFO 26087 --- [ocp-schedules-9,c9a2d950e080e247,ffc27705a4070bca] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:11:17.232 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:17.233 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:17.449 INFO 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:11:17.450 INFO 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:11:17.450 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:11:17.450 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:11:17.450 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:11:17.452 INFO 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:11:17.453 WARN 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:11:17.454 INFO 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:11:17.454 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:11:17.454 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:11:17.454 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:11:17.455 WARN 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:11:17.455 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:11:17.455 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:11:17.455 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:11:17.455 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:11:17.456 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:11:17.456 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:11:17.456 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:11:17.456 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:11:17.457 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:11:17.458 INFO 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:11:17.460 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:11:17.460 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:11:17.460 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:11:17.460 WARN 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:11:17.461 ERROR 26087 --- [ocp-schedules-29,90d0f99ef5e13b1a,1b086e4a95ef1169] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.62:62889 [/192.168.1.62] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.62:62889 [/192.168.1.62] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 21:11:18.355 INFO 26087 --- [prometheus-minute-schedule-1,2c7ea623e1e6d55d,51ad81bf47217e3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:18.472 INFO 26087 --- [ocp-schedules-1,2e5ae99c5f77f4c2,789f7913ea509d60] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:11:18.581 WARN 26087 --- [ocp-async-6,f636a65025575644,cc3a003fceaa1e30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:18.581 INFO 26087 --- [ocp-async-7,df066a226609094f,7d11eeca34e5d8b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:20.233 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:20.234 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:20.581 WARN 26087 --- [ocp-async-7,075bca8ff75794d8,fd2f58702f9a64ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:20.581 INFO 26087 --- [ocp-async-1,bc45b9d19e139913,c7721d57e9b9f8b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:11:21.355 INFO 26087 --- [prometheus-minute-schedule-1,ca5ef378d827b9f1,8b1807fca57e08ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:22.581 WARN 26087 --- [ocp-async-3,3acc80506dd568ad,bb17817518631d18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:22.581 INFO 26087 --- [ocp-async-1,d609299b94085f91,6f0425b0963702a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:23.234 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:23.234 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:24.355 INFO 26087 --- [prometheus-minute-schedule-1,c0529b5948fbdb25,70b68bdd033165a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:24.582 WARN 26087 --- [ocp-async-6,e33c23e2d93c9f56,f6893f9e9a6e488d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:24.582 INFO 26087 --- [ocp-async-8,eed190fac5367dd4,f5b1dc21ce480fdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:26.234 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:26.235 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:26.526 INFO 26087 --- [ocp-schedules-21,4999daa259233cf0,3302aa3f81dc0041] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:11:26.526 INFO 26087 --- [ocp-schedules-18,25f44d629c80a957,b65ec4db7af51b16] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:11:26.545 INFO 26087 --- [ocp-async-7,6b41cfe2926f0801,6674762ff8e3d739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:11:26.582 INFO 26087 --- [ocp-async-10,b16b971f2fcf3950,9b45bec5850fe224] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:26.582 WARN 26087 --- [ocp-async-2,e61f25f1b73ec835,fce2ed5b40b246fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:26.613 INFO 26087 --- [ocp-schedules-4,479dcc7d832df745,de3a1f811a23fb00] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:11:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:11:26.622384+08:00, lastProcessTime=2025-04-08T21:11:16.622978+08:00 2025-04-08 21:11:26.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:11:26.622384+08:00 2025-04-08 21:11:27.219 INFO 26087 --- [ocp-schedules-3,de2ab5ff00c7450c,93717629486eeb6e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:11:27.220 INFO 26087 --- [ocp-schedules-3,de2ab5ff00c7450c,93717629486eeb6e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:11:27.355 INFO 26087 --- [prometheus-minute-schedule-1,55a73f5f48482554,d079157001b8dc5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:28.505 INFO 26087 --- [ocp-schedules-2,9646b6843ca3b9bc,942c4aa3dd22e375] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:11:28.582 WARN 26087 --- [ocp-async-6,c74b879d8367b37c,0efa6783b1d58bca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:28.582 INFO 26087 --- [ocp-async-9,983945f2fe111616,ba0a5a5b311ad1fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:29.235 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:29.236 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:30.355 INFO 26087 --- [prometheus-minute-schedule-1,290a0415b9ceaaf8,cf351ee4f9b267e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:30.582 WARN 26087 --- [ocp-async-5,b7653507aaf1b29e,b109e375ae27e9d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:30.582 INFO 26087 --- [ocp-async-4,df962aaa2e259630,80b352821a508509] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:32.236 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:32.236 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:32.582 WARN 26087 --- [ocp-async-7,743f83ea2f36a046,d0d7f17ba3e9d4f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:32.582 INFO 26087 --- [ocp-async-2,17f5056e12b55937,4b9ebe56ff73248e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:33.355 INFO 26087 --- [prometheus-minute-schedule-1,9cae6f31eecd5d03,ede32346d4fa3953] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:34.582 WARN 26087 --- [ocp-async-6,cf6d60a6678c7d91,323571f397a8b014] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:34.582 INFO 26087 --- [ocp-async-10,75fdeea46e326178,760693698b084f60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:34.978 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:11:34.984 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster sgkydb(1000002), status=TAKINGOVER 2025-04-08 21:11:34.985 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=6ms, period=120000ms 2025-04-08 21:11:35.236 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:35.237 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:36.355 INFO 26087 --- [prometheus-minute-schedule-1,4ab0d2a9ea7b63a4,bee6988b374ee532] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:36.545 INFO 26087 --- [ocp-async-10,58725aaee9c35130,b15b475cc8089bd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:11:36.582 WARN 26087 --- [ocp-async-8,ea1e54bdbe2dcd62,1ea97610f1e71302] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:36.583 INFO 26087 --- [ocp-async-5,8088fe5e0dfaa8dd,8844c6c31201c1f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:36.615 INFO 26087 --- [ocp-schedules-28,aabed03cede5faca,065486070d7ece1a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:11:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:11:36.623800+08:00, lastProcessTime=2025-04-08T21:11:26.622384+08:00 2025-04-08 21:11:36.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:11:36.623800+08:00 2025-04-08 21:11:36.961 INFO 26087 --- [ocp-schedules-24,b57c1b440947c8d4,a6a8eff4d35f4614] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:11:36.977 INFO 26087 --- [ocp-schedules-24,b57c1b440947c8d4,a6a8eff4d35f4614] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:11:36.977 INFO 26087 --- [ocp-schedules-24,b57c1b440947c8d4,a6a8eff4d35f4614] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:11:37.226 INFO 26087 --- [ocp-schedules-14,726019ac56c75083,241489b42ba56a80] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:11:37.227 INFO 26087 --- [ocp-schedules-14,726019ac56c75083,241489b42ba56a80] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:11:38.237 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:38.237 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:38.527 INFO 26087 --- [ocp-schedules-30,9d8a0c80c28047a3,23e249e02524a333] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:11:38.583 WARN 26087 --- [ocp-async-7,34b026682c79fca5,8f1af7844c6c9f50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:38.583 INFO 26087 --- [ocp-async-3,e051ce5ebe220570,fe448b73ea38090f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:39.355 INFO 26087 --- [prometheus-minute-schedule-1,e9912b3a9a1658ca,4df0c421fa9b2106] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:40.002 INFO 26087 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:11:40.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:11:40.583 WARN 26087 --- [ocp-async-3,e574bdaf23a18364,4a9ed489000a05b4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:40.583 INFO 26087 --- [ocp-async-1,6b8eb5a35de8fb01,e1c79a0813afaeca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:41.237 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:41.238 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:42.355 INFO 26087 --- [prometheus-minute-schedule-1,afaa8c49270e91a3,9069e443161607b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:42.583 WARN 26087 --- [ocp-async-10,c8bf1596a204e3fb,82af04452586a7c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:42.583 INFO 26087 --- [ocp-async-1,77b4a55dd9294e36,28cda6e8eee7c703] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:44.238 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:44.239 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:44.583 WARN 26087 --- [ocp-async-2,c4aa666da790fb49,6020a9827e822708] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:44.583 INFO 26087 --- [ocp-async-1,5b43937519b0b0ec,c5620fbf4f5916a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:45.355 INFO 26087 --- [prometheus-minute-schedule-1,e697cde4f9269f00,9bd581b47b1bf5f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:46.521 INFO 26087 --- [ocp-schedules-23,d81beb0013c1ee00,b241c4bad93c6b07] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:11:46.521 INFO 26087 --- [ocp-schedules-2,d9b9750ecc079523,99276a214c9df399] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:11:46.521 INFO 26087 --- [ocp-schedules-2,d9b9750ecc079523,99276a214c9df399] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:11:46.523 INFO 26087 --- [ocp-schedules-17,05b9965c90c1c0a5,21dac396b3f21784] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:11:46.523 INFO 26087 --- [ocp-schedules-17,05b9965c90c1c0a5,21dac396b3f21784] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:11:46.524 INFO 26087 --- [ocp-schedules-20,6240fdaa52e12cf3,6f3da2b8b400f55e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:11:46.524 INFO 26087 --- [ocp-schedules-20,6240fdaa52e12cf3,6f3da2b8b400f55e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:11:46.524 INFO 26087 --- [ocp-schedules-18,fdd2247bf0049269,a0116032b7d38bb2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:11:46.524 INFO 26087 --- [ocp-schedules-21,dde1b51b6432f885,845687d9efca2433] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:11:46.525 INFO 26087 --- [ocp-schedules-18,fdd2247bf0049269,a0116032b7d38bb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:46.525 INFO 26087 --- [ocp-schedules-21,dde1b51b6432f885,845687d9efca2433] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:11:46.534 INFO 26087 --- [ocp-schedules-8,b0791d6f24414ebb,4ba44e2e1a1334a7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:11:46.545 INFO 26087 --- [ocp-async-7,05849c647fdb8f45,8abbece2cc0ee0a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:11:46.550 INFO 26087 --- [ocp-schedules-16,be1c29b362e586fd,74d29eb0d980e258] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:11:46.550 INFO 26087 --- [ocp-schedules-16,06b7cedffcea81b9,3953ff41384a974c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:11:46.551 INFO 26087 --- [ocp-schedules-16,06b7cedffcea81b9,3953ff41384a974c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:11:46.558 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:11:46.558 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:11:46.583 INFO 26087 --- [ocp-async-3,3de951aa02fe2ef6,682743bb7a61c8ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:46.584 WARN 26087 --- [ocp-async-9,36e173a421d3ea1c,ede982740e7f4bb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:46.591 INFO 26087 --- [ocp-schedules-6,7d080ae3da560822,9f6de5b46dc52668] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:11:46.616 INFO 26087 --- [ocp-schedules-3,9a313e85a7c5baf3,6942b4073d1a322d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:11:46.622 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:11:46.622428+08:00, lastProcessTime=2025-04-08T21:11:36.623800+08:00 2025-04-08 21:11:46.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:11:46.622428+08:00 2025-04-08 21:11:47.231 INFO 26087 --- [ocp-schedules-12,348754898459b468,f23ad6e3f6e630f2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:11:47.232 INFO 26087 --- [ocp-schedules-12,348754898459b468,f23ad6e3f6e630f2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:11:47.239 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:47.239 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:48.355 INFO 26087 --- [prometheus-minute-schedule-1,2a8fbb19553cddf8,0d83edb281c8166b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:48.559 INFO 26087 --- [ocp-schedules-5,754e73dd0b3f10bf,6440eaa272c925eb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:11:48.584 INFO 26087 --- [ocp-async-3,64a97aa11b0c52d0,504453ecd5a35e0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:48.584 WARN 26087 --- [ocp-async-5,963bcabdde5ccd0d,2b5756e70e703891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:50.239 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:50.240 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:50.584 WARN 26087 --- [ocp-async-1,4cd6c233bb131b04,6ba736bd73b7c442] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:50.584 INFO 26087 --- [ocp-async-2,7cc68b198f0fbeb7,1bba3860b750ec0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:51.355 INFO 26087 --- [prometheus-minute-schedule-1,f7fcf984eb82373a,aaa2d6ecd24b9782] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:52.584 WARN 26087 --- [ocp-async-8,1e1746e03f19b732,7168ef52c2c334a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:52.584 INFO 26087 --- [ocp-async-7,6e05bad6d12987c0,ca8049daf51f9f7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:53.240 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:53.241 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:54.355 INFO 26087 --- [prometheus-minute-schedule-1,e6dbf956e58e5034,6f7bcc889e91a48d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:54.584 WARN 26087 --- [ocp-async-10,ebf78d18ab70227d,4bc984937dd5ed0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:54.584 INFO 26087 --- [ocp-async-4,e24f806ee30b6d86,6ae85a061832bac9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:56.241 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:56.242 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:11:56.526 INFO 26087 --- [ocp-schedules-18,fdd2247bf0049269,a0116032b7d38bb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:11:56.526 INFO 26087 --- [ocp-schedules-21,dde1b51b6432f885,845687d9efca2433] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:11:56.545 INFO 26087 --- [ocp-async-4,7834ddba8af8fd78,72b670e05eea4457] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:11:56.584 WARN 26087 --- [ocp-async-6,5ebc8386db63d899,23149905dfe645a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:56.584 INFO 26087 --- [ocp-async-1,a5b27b8d78c8d06d,28d59cd829dd85b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:56.618 INFO 26087 --- [ocp-schedules-22,43a8d2c9cc059f27,2c7392fac926f91d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:11:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:11:56.623614+08:00, lastProcessTime=2025-04-08T21:11:46.622428+08:00 2025-04-08 21:11:56.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:11:56.623614+08:00 2025-04-08 21:11:56.978 INFO 26087 --- [ocp-schedules-27,72631806f1625743,bb3432602ea91311] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:11:56.984 INFO 26087 --- [ocp-schedules-27,72631806f1625743,bb3432602ea91311] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:11:56.984 INFO 26087 --- [ocp-schedules-27,72631806f1625743,bb3432602ea91311] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:11:57.237 INFO 26087 --- [ocp-schedules-23,fe4d3815e9d3c5a5,7287c7952e183c83] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:11:57.238 INFO 26087 --- [ocp-schedules-23,fe4d3815e9d3c5a5,7287c7952e183c83] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:11:57.355 INFO 26087 --- [prometheus-minute-schedule-1,19c3407dec81b906,cd03ce356a28b704] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117920, interval=60 2025-04-08 21:11:58.584 WARN 26087 --- [ocp-async-7,9e48d0cc162166c4,43a08b3ed7a5ef91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:11:58.584 INFO 26087 --- [ocp-async-9,83a78503d65f1c98,1ea5dab1617801da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:11:58.585 INFO 26087 --- [ocp-schedules-1,d94ca80712825f10,31ee7750a72f7f84] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:11:59.242 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:11:59.242 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:00.355 INFO 26087 --- [prometheus-minute-schedule-1,7a0b5d6a3a9b7028,799548fb9c971cc2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:00.584 WARN 26087 --- [ocp-async-5,4a88f4d7a6c375cd,cc673902bc071fd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:00.584 INFO 26087 --- [ocp-async-10,1e7d75dfac61daf2,4295715814eed80a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:02.243 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:02.243 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:02.585 WARN 26087 --- [ocp-async-6,e6a4bd38b3863d6d,dd9ef304a7880489] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:02.585 INFO 26087 --- [ocp-async-2,420b9b4cb2dc0d8e,a9463f66e4dab50d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:03.355 INFO 26087 --- [prometheus-minute-schedule-1,285de0a2d0834b2c,cf9ad259c3b6815c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:04.585 WARN 26087 --- [ocp-async-2,d60ec0e1de51633b,d8aa71a0def7a0c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:04.585 INFO 26087 --- [ocp-async-7,5aa18941ff4bb036,b566e4709ba9f6e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:05.243 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:05.244 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:06.347 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:08:06.292, end=2025-04-08T21:10:06.344} 2025-04-08 21:12:06.349 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:12:06.349 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:12:06.355 INFO 26087 --- [prometheus-minute-schedule-1,308f177aa258479c,f9cd78e03c4cae1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:06.355 INFO 26087 --- [prometheus-cache-monitor1,1ac438daf5636430,4926db0462c6dfbd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1494,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:12:06.355 INFO 26087 --- [prometheus-cache-monitor1,1ac438daf5636430,4926db0462c6dfbd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:12:06.546 INFO 26087 --- [ocp-async-7,13194a6b57b632a2,a74aa0b878a97f3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:12:06.561 INFO 26087 --- [ocp-schedules-14,a0b882552b4bd406,08e39781938f48d4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:12:06.562 INFO 26087 --- [ocp-schedules-14,a0b882552b4bd406,08e39781938f48d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:06.562 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:12:06.585 WARN 26087 --- [ocp-async-9,5c70bd5c80aa4a3d,aac658b7006d671f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:06.585 INFO 26087 --- [ocp-async-3,76ea34c0e8b192e2,86cc4c92ba06b8cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:06.619 INFO 26087 --- [ocp-schedules-30,6d0f0086b68b41b7,ce9b71513c6772fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:12:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:12:06.624036+08:00, lastProcessTime=2025-04-08T21:11:56.623614+08:00 2025-04-08 21:12:06.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:12:06.624036+08:00 2025-04-08 21:12:07.247 INFO 26087 --- [ocp-schedules-15,55417f4fd6515391,5dcf62a2cf5b03f7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:12:07.248 INFO 26087 --- [ocp-schedules-15,55417f4fd6515391,5dcf62a2cf5b03f7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:12:08.244 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:08.245 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:08.585 WARN 26087 --- [ocp-async-4,0c6acb11deb36ab4,8e7c82d61d4186f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:08.585 INFO 26087 --- [ocp-async-6,a167794c34302a3d,1d90213847ba4ecc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:08.615 INFO 26087 --- [ocp-schedules-11,dc93b19007fd2331,fcaebb889f133191] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:12:09.355 INFO 26087 --- [prometheus-minute-schedule-1,7e3149db00a6f134,e607fd58258fec94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:10.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:12:10.002 INFO 26087 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:12:10.585 WARN 26087 --- [ocp-async-8,fcdd6ad14001b3c8,1863ad8416a7f509] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:10.585 INFO 26087 --- [ocp-async-1,5b15566065f1265e,eda7a02615703626] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:10.601 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:12:11.245 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:11.245 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:12.355 INFO 26087 --- [prometheus-minute-schedule-1,7fb7ace7fa1bb471,6d4e87ef9debdeb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:12.397 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117860, nodeCount=19, affectRows=19 2025-04-08 21:12:12.397 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117860, nodeCount=20, affectRows=20 2025-04-08 21:12:12.397 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117860, nodeCount=20, validCount=20 2025-04-08 21:12:12.397 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117860, nodeCount=19, validCount=19 2025-04-08 21:12:12.402 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117860, nodeCount=91, affectRows=91 2025-04-08 21:12:12.402 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117860, nodeCount=105, affectRows=105 2025-04-08 21:12:12.402 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117860, nodeCount=115, affectRows=115 2025-04-08 21:12:12.402 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117860, nodeCount=24, affectRows=24 2025-04-08 21:12:12.402 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117860, nodeCount=24, validCount=24 2025-04-08 21:12:12.403 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117860, nodeCount=90, validCount=90 2025-04-08 21:12:12.403 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117860, nodeCount=104, validCount=104 2025-04-08 21:12:12.403 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117860, nodeCount=114, validCount=114 2025-04-08 21:12:12.585 WARN 26087 --- [ocp-async-10,fc4b9535e3c32764,a8db5e830501e4e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:12.585 INFO 26087 --- [ocp-async-1,c86a008f8f44a673,88417b7ea8d497d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:14.245 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:14.246 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:14.585 WARN 26087 --- [ocp-async-1,afb0919d05c3d1ec,bb26a431ffcaefa1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:14.586 INFO 26087 --- [ocp-async-4,88f0f70d37391c95,fad217149d1e3331] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:15.355 INFO 26087 --- [prometheus-minute-schedule-1,1ca923911cc13b79,04c0e14e9569d3b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:16.517 INFO 26087 --- [ocp-schedules-11,eb52a9802253b282,32eb20fa8f0bd139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:12:16.517 INFO 26087 --- [ocp-schedules-17,b1d21acf1581d0ed,d28f29f2c29b1a3d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:12:16.521 INFO 26087 --- [ocp-schedules-18,5def5c863a0f47db,4d1912f738bd80fb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:12:16.522 INFO 26087 --- [ocp-schedules-5,f1d4036a6f028701,52d908dd5fc9713d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:12:16.522 INFO 26087 --- [ocp-schedules-5,f1d4036a6f028701,52d908dd5fc9713d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:12:16.523 INFO 26087 --- [ocp-schedules-23,90596252da507637,fdae5015fac2f202] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:12:16.523 INFO 26087 --- [ocp-schedules-23,90596252da507637,fdae5015fac2f202] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:12:16.524 INFO 26087 --- [ocp-schedules-8,36e9bfa65348a18f,355463ed10dab098] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:12:16.524 INFO 26087 --- [ocp-schedules-8,36e9bfa65348a18f,355463ed10dab098] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:12:16.524 INFO 26087 --- [ocp-schedules-17,b1d21acf1581d0ed,d28f29f2c29b1a3d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:12:16.524 INFO 26087 --- [ocp-schedules-29,f8d9dc66881ef70b,d2885e5a5c58edfe] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:12:16.524 INFO 26087 --- [ocp-schedules-15,802b057699470e8e,68d9b554edc7aa86] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:12:16.525 INFO 26087 --- [ocp-schedules-29,f8d9dc66881ef70b,d2885e5a5c58edfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:16.525 INFO 26087 --- [ocp-schedules-15,802b057699470e8e,68d9b554edc7aa86] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:12:16.529 INFO 26087 --- [ocp-schedules-11,eb52a9802253b282,32eb20fa8f0bd139] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:12:16.546 INFO 26087 --- [ocp-async-6,28fe2be7bd395429,240baf12c03c723c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:12:16.558 INFO 26087 --- [ocp-schedules-16,06b7cedffcea81b9,3953ff41384a974c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:12:16.558 INFO 26087 --- [ocp-schedules-16,d8b50189c31325fd,afd7c9e502215746] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:12:16.559 INFO 26087 --- [ocp-schedules-16,d8b50189c31325fd,afd7c9e502215746] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:16.562 INFO 26087 --- [ocp-schedules-14,a0b882552b4bd406,08e39781938f48d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:12:16.565 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:12:16.565 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:12:16.582 INFO 26087 --- [ocp-schedules-6,272b0ca593e684ff,a0602e231a3b96a5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:12:16.586 WARN 26087 --- [ocp-async-9,2d00ad7c5d57b365,94c3cf03f48f2739] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:16.586 INFO 26087 --- [ocp-async-10,0a6150880c682e68,2ae70fb21c10c897] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:16.592 INFO 26087 --- [ocp-schedules-26,c5c8b5b0d03a7e23,e4d4e18f15d8889e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:12:16.602 INFO 26087 --- [ocp-schedules-7,a6817a8366678ff9,acfceafaf9c576b2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:12:16.607 INFO 26087 --- [ocp-schedules-9,c40b000aa14adf57,b0f9145b3764d3a2] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:12:16.607 INFO 26087 --- [ocp-schedules-9,c40b000aa14adf57,b0f9145b3764d3a2] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:12:16.610 INFO 26087 --- [ocp-schedules-23,7caec3d54784eda5,e8397180d66c39a9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:12:16.610+08:00, clusters=[1000002] 2025-04-08 21:12:16.610 INFO 26087 --- [ocp-schedules-23,7caec3d54784eda5,e8397180d66c39a9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:12:16.610 INFO 26087 --- [ocp-schedules-8,f641f3b09a800259,8c3a337151e07a66] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:12:16.611 INFO 26087 --- [ocp-schedules-17,a711d40da91f0a55,93b72f13277777f5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:12:16.611+08:00, clusters=[1000002] 2025-04-08 21:12:16.613 INFO 26087 --- [ocp-schedules-17,a711d40da91f0a55,93b72f13277777f5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:12:16.620 INFO 26087 --- [ocp-schedules-30,130a5061fb1da304,57058750693d2cc2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:12:16.622 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:12:16.622048+08:00, lastProcessTime=2025-04-08T21:12:06.624036+08:00 2025-04-08 21:12:16.630 INFO 26087 --- [ocp-schedules-25,a1a795b48ca28934,2ec63ad459a14034] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:12:16.630 INFO 26087 --- [ocp-schedules-25,a1a795b48ca28934,2ec63ad459a14034] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:12:16.632 INFO 26087 --- [ocp-schedules-3,35b3a3394ec6918f,438611cb01adc337] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:12:16.632 INFO 26087 --- [ocp-schedules-3,35b3a3394ec6918f,438611cb01adc337] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:12:16.638 INFO 26087 --- [ocp-schedules-3,35b3a3394ec6918f,438611cb01adc337] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:12:16.638 INFO 26087 --- [ocp-schedules-3,35b3a3394ec6918f,438611cb01adc337] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:12:16.659 INFO 26087 --- [ocp-schedules-28,77ddf8d1fa13d4d1,fab788be5dba9bea] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:12:16.661 INFO 26087 --- [ocp-schedules-28,77ddf8d1fa13d4d1,fab788be5dba9bea] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:12:16.661 INFO 26087 --- [ocp-schedules-28,77ddf8d1fa13d4d1,fab788be5dba9bea] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:12:16.695 INFO 26087 --- [ocp-schedules-1,460a64cdfbe78737,1f00af4d82ad0b17] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:12:16.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:12:16.622048+08:00 2025-04-08 21:12:16.727 INFO 26087 --- [ocp-schedules-13,e553bbd0ca9c396d,5f3de561f1fcddb3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:12:16.727 INFO 26087 --- [ocp-schedules-13,e553bbd0ca9c396d,5f3de561f1fcddb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:16.745 INFO 26087 --- [ocp-schedules-21,3b78ed4e1e0035a1,113fbc7a8a3286a4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:12:16.746 INFO 26087 --- [ocp-schedules-21,3b78ed4e1e0035a1,113fbc7a8a3286a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:16.747 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:12:16.748 ERROR 26087 --- [ocp-schedules-13,e553bbd0ca9c396d,5f3de561f1fcddb3] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:12:16.748 INFO 26087 --- [ocp-schedules-13,e553bbd0ca9c396d,5f3de561f1fcddb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:12:16.756 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:12:16.757 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:12:16.817 INFO 26087 --- [ocp-schedules-27,456f66abe3354bb0,fd1203287b22e167] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:12:16.817 INFO 26087 --- [ocp-schedules-27,456f66abe3354bb0,fd1203287b22e167] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:12:16.818 INFO 26087 --- [ocp-schedules-27,456f66abe3354bb0,fd1203287b22e167] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:12:16.818 INFO 26087 --- [ocp-schedules-27,456f66abe3354bb0,fd1203287b22e167] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208338840276992 2025-04-08 21:12:16.818 INFO 26087 --- [ocp-schedules-27,456f66abe3354bb0,fd1203287b22e167] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:12:16.818 INFO 26087 --- [ocp-schedules-27,456f66abe3354bb0,fd1203287b22e167] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:12:16.985 INFO 26087 --- [ocp-schedules-14,ccfcf97f47e6dde5,35f4d10a4f917f44] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:12:16.989 INFO 26087 --- [ocp-schedules-14,ccfcf97f47e6dde5,35f4d10a4f917f44] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:12:16.989 INFO 26087 --- [ocp-schedules-14,ccfcf97f47e6dde5,35f4d10a4f917f44] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:12:17.246 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:17.247 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:17.257 INFO 26087 --- [ocp-schedules-5,21cde9ce918d9728,6f24ffcfd27f923f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:12:17.258 INFO 26087 --- [ocp-schedules-5,21cde9ce918d9728,6f24ffcfd27f923f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:12:17.464 INFO 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:12:17.464 INFO 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:12:17.464 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:12:17.464 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:12:17.464 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:12:17.467 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:12:17.467 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:12:17.467 INFO 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:12:17.467 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:12:17.468 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:12:17.468 WARN 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:12:17.468 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:12:17.468 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:12:17.468 INFO 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:12:17.468 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:12:17.468 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:12:17.468 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:12:17.469 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:12:17.469 WARN 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:12:17.469 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:12:17.469 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:12:17.472 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:12:17.472 INFO 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:12:17.472 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:12:17.473 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:12:17.474 WARN 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:12:17.475 ERROR 26087 --- [ocp-schedules-23,b0576b96867d8fd7,ce29197929d599fa] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.62:62889 [/192.168.1.62] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.62:62889 [/192.168.1.62] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 21:12:18.355 INFO 26087 --- [prometheus-minute-schedule-1,3b401183dbe4c14f,ed44b8246943833a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:18.586 WARN 26087 --- [ocp-async-3,9d8539268519783e,4eaae0afd5615abb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:18.586 INFO 26087 --- [ocp-async-10,edee931e718f5836,956656990d1adaee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:18.642 INFO 26087 --- [ocp-schedules-18,8d2981978a31ec13,02ddd8367f684191] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:12:20.247 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:20.248 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:20.498 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:12:20.498 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:12:20.502 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:12:20.502 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:12:20.502 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:12:20.502 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:12:20.505 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:12:20.505 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:12:20.505 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:12:20.506 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:12:20.528 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:12:20.533 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:12:20.533 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:12:20.533 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:12:20.533 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:12:20.533 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:12:20.539 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:12:20.586 WARN 26087 --- [ocp-async-4,b70549834059e59d,2f898823e0c40d14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:20.586 INFO 26087 --- [ocp-async-8,4375a1e98af28c38,fe7f7a49f2c50811] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:12:20.850 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:12:21.214 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.708055591S 2025-04-08 21:12:21.214 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:12:21.271 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:12:21.271 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:12:21.297 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:12:21.339 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:12:21.339 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:12:21.339 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:12:21.339 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:12:21.339 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:12:21.339 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:12:21.347 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:12:21.355 INFO 26087 --- [prometheus-minute-schedule-1,f3094f4d8266223d,a4ed0ecdd320a052] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:21.453 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] o.s.boot.SpringApplication : Started application in 0.95 seconds (JVM running for 646.826) 2025-04-08 21:12:21.453 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.951139166S 2025-04-08 21:12:21.707 INFO 26087 --- [ocp-schedules-20,f6e644ff41403919,7529aa4cb5d1880a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:12:22.586 WARN 26087 --- [ocp-async-5,43af03b4e0e2c36e,d83299edd7d4917c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:22.586 INFO 26087 --- [ocp-async-7,aa239463fbc1b47d,c4b01c93b5724408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:23.248 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:23.248 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:24.355 INFO 26087 --- [prometheus-minute-schedule-1,261708fde188d31c,3c049159203e7619] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:24.586 WARN 26087 --- [ocp-async-7,2195e0abd8340677,a2de6596657a98e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:24.586 INFO 26087 --- [ocp-async-2,7d20837a089dbe99,cb2959ed902155c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:26.248 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:26.249 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:26.525 INFO 26087 --- [ocp-schedules-15,802b057699470e8e,68d9b554edc7aa86] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:12:26.525 INFO 26087 --- [ocp-schedules-29,f8d9dc66881ef70b,d2885e5a5c58edfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:12:26.546 INFO 26087 --- [ocp-async-4,fbe41f4a13cd2aac,4c8bb930f4c15c39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:12:26.586 WARN 26087 --- [ocp-async-8,a5a13d8bf387ebaf,f680dc42562c406a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:26.586 INFO 26087 --- [ocp-async-5,3f0bc97db6457bc3,036d2d486984cecd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:26.621 INFO 26087 --- [ocp-schedules-19,53e40d73ffd0e1ab,00cbfa3128e649dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:12:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:12:26.623561+08:00, lastProcessTime=2025-04-08T21:12:16.622048+08:00 2025-04-08 21:12:26.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:12:26.623561+08:00 2025-04-08 21:12:27.262 INFO 26087 --- [ocp-schedules-13,43a4df270085f2aa,e3fb772003bacd4b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:12:27.264 INFO 26087 --- [ocp-schedules-13,43a4df270085f2aa,e3fb772003bacd4b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:12:27.355 INFO 26087 --- [prometheus-minute-schedule-1,eba0f17a4245eb54,b69122b813fa2e12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:28.586 WARN 26087 --- [ocp-async-5,cc7115ba96f7ba9e,dc5bd7eedde87e6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:28.587 INFO 26087 --- [ocp-async-7,55491da063b3dab8,418308066735abfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:28.666 INFO 26087 --- [ocp-schedules-22,f00cba25c0329b2b,c253ee2d5173704e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:12:29.249 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:29.250 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:30.355 INFO 26087 --- [prometheus-minute-schedule-1,e4c3848dd451cd13,de4d1dc9f338d19e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:30.587 WARN 26087 --- [ocp-async-1,7f2e8a74ddd18387,900a6ecd1ed68f3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:30.587 INFO 26087 --- [ocp-async-2,6126d2a69f0fe1e7,32b7e0118891267b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:32.250 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:32.251 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:32.587 WARN 26087 --- [ocp-async-8,e07d7d9444ef07ef,e4027187ca0c31de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:32.587 INFO 26087 --- [ocp-async-10,ff336e9e094160fa,8a50dd6f1a376c76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:33.355 INFO 26087 --- [prometheus-minute-schedule-1,207c06feb661addd,3c962195e5740c3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:34.587 WARN 26087 --- [ocp-async-6,342faeba44f46638,988e5d33d1876164] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:34.587 INFO 26087 --- [ocp-async-5,d7b4b4ea35bd858d,6cc494ddc06c1923] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:35.251 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:35.251 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:36.355 INFO 26087 --- [prometheus-minute-schedule-1,1484a0acc2c47cd8,90f5adb50ee525a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:36.546 INFO 26087 --- [ocp-async-1,4c456d9f5e46f75e,7e033b5792211ced] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:12:36.587 WARN 26087 --- [ocp-async-3,ac79621de7049a6b,22229c267614eb5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:36.587 INFO 26087 --- [ocp-async-2,4dedd564e9de2362,ba27dacea453810a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:36.622 INFO 26087 --- [ocp-schedules-18,d5f4e20ca902aad4,1e61058e0204a1c3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:12:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:12:36.623956+08:00, lastProcessTime=2025-04-08T21:12:26.623561+08:00 2025-04-08 21:12:36.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:12:36.623956+08:00 2025-04-08 21:12:36.990 INFO 26087 --- [ocp-schedules-23,7e32ce4e9c605ee6,c59ecf465be40841] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:12:36.997 INFO 26087 --- [ocp-schedules-23,7e32ce4e9c605ee6,c59ecf465be40841] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:12:36.997 INFO 26087 --- [ocp-schedules-23,7e32ce4e9c605ee6,c59ecf465be40841] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:12:37.268 INFO 26087 --- [ocp-schedules-20,fe6d65cb8c5a5083,6453d62c77adf090] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:12:37.269 INFO 26087 --- [ocp-schedules-20,fe6d65cb8c5a5083,6453d62c77adf090] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:12:38.251 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:38.252 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:38.587 WARN 26087 --- [ocp-async-4,4f7bf5ed980e9cea,36cf2dd8cbc13fb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:38.587 INFO 26087 --- [ocp-async-8,ae630ab48da3b012,470d006a562ffbef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:38.687 INFO 26087 --- [ocp-schedules-4,4ba72765071ad05c,aa6d8562a72763bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:12:39.355 INFO 26087 --- [prometheus-minute-schedule-1,69a1523b046d2f4c,3a3ac3ef92c86103] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:40.001 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:12:40.001 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:12:40.587 WARN 26087 --- [ocp-async-7,3250ec7dcbb10875,dc89df0fc8dc06b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:40.587 INFO 26087 --- [ocp-async-8,bf69c3eaa06115cb,93e751f48ff25069] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:41.252 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:41.253 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:42.355 INFO 26087 --- [prometheus-minute-schedule-1,7f4b1c25a8ce5d95,71963c00ba310c6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:42.587 WARN 26087 --- [ocp-async-1,2e0d8aac5a401de3,65326c0a743807d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:42.587 INFO 26087 --- [ocp-async-3,fa196dd837d7f39d,5ecd3bda374d149f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:44.253 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:44.253 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:44.588 WARN 26087 --- [ocp-async-3,704baca9fb9cd25e,946e7ab319c0012a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:44.588 INFO 26087 --- [ocp-async-4,4981c0dfdc89cb03,9ffe0d62e50aae8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:45.355 INFO 26087 --- [prometheus-minute-schedule-1,18c4aecc9c124f25,9a8965d774393816] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:46.521 INFO 26087 --- [ocp-schedules-8,82e4e8f41d0703ba,c680137a8a6223f6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:12:46.523 INFO 26087 --- [ocp-schedules-9,bc35688ce8769e5b,529cb0b288c6fafc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:12:46.523 INFO 26087 --- [ocp-schedules-9,bc35688ce8769e5b,529cb0b288c6fafc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:12:46.524 INFO 26087 --- [ocp-schedules-22,0a03de11c356a18b,ca1141d94599b939] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:12:46.524 INFO 26087 --- [ocp-schedules-29,78e7c36bc153fc39,66c49431940b47af] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:12:46.524 INFO 26087 --- [ocp-schedules-25,8aba0d49d24a75dc,c3ce7ee7a3fdf9a0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:12:46.524 INFO 26087 --- [ocp-schedules-7,9911b6ad8d98931d,6754731f826e763d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:12:46.525 INFO 26087 --- [ocp-schedules-7,9911b6ad8d98931d,6754731f826e763d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:12:46.525 INFO 26087 --- [ocp-schedules-4,15720aafffd0c26c,d095858705e01dce] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:12:46.525 INFO 26087 --- [ocp-schedules-4,15720aafffd0c26c,d095858705e01dce] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:12:46.525 INFO 26087 --- [ocp-schedules-29,78e7c36bc153fc39,66c49431940b47af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:46.525 INFO 26087 --- [ocp-schedules-25,8aba0d49d24a75dc,c3ce7ee7a3fdf9a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:12:46.546 INFO 26087 --- [ocp-async-4,9bcdb823154baf65,999b72c350aedeed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:12:46.565 INFO 26087 --- [ocp-schedules-16,d8b50189c31325fd,afd7c9e502215746] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:12:46.565 INFO 26087 --- [ocp-schedules-16,501ae53efe84cfc3,b91330ecb26953f3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:12:46.566 INFO 26087 --- [ocp-schedules-16,501ae53efe84cfc3,b91330ecb26953f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:12:46.572 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:12:46.572 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:12:46.588 WARN 26087 --- [ocp-async-10,57b1bc0f71a901ae,7dcab27caeb925db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:46.588 INFO 26087 --- [ocp-async-8,1a9401f7144adbdd,a30cc31d1a4d5b7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:46.593 INFO 26087 --- [ocp-schedules-10,a1bab9fcf4b4248c,4ccbb5e732d51267] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:12:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:12:46.622580+08:00, lastProcessTime=2025-04-08T21:12:36.623956+08:00 2025-04-08 21:12:46.624 INFO 26087 --- [ocp-schedules-19,077aac3c2b04ac6c,ba4a5f267cb0a460] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:12:46.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:12:46.622580+08:00 2025-04-08 21:12:47.254 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:47.254 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:47.275 INFO 26087 --- [ocp-schedules-28,91024131369937cd,826991aeeb8703ae] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:12:47.276 INFO 26087 --- [ocp-schedules-28,91024131369937cd,826991aeeb8703ae] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:12:48.355 INFO 26087 --- [prometheus-minute-schedule-1,86de8d747610f58b,c3d17447492701b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:48.588 WARN 26087 --- [ocp-async-5,d74a59ec63b4d290,0590a07dda32c540] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:48.588 INFO 26087 --- [ocp-async-2,3c4a168b13ff93e6,40003b2f8689bf7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:48.715 INFO 26087 --- [ocp-schedules-12,159ed2767288a548,9aaa315dfaed7f2a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:12:50.254 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:50.255 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:50.588 WARN 26087 --- [ocp-async-9,684e0de602469653,d8bdf649a75f74b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:50.588 INFO 26087 --- [ocp-async-6,62be015c54c2ebbf,9a4b7c7307bab896] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:51.355 INFO 26087 --- [prometheus-minute-schedule-1,770c66c7e3e92d3f,f2ea05f134ff0365] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:52.588 WARN 26087 --- [ocp-async-7,9d0895f880c1cfea,c1b154b875ae79fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:52.588 INFO 26087 --- [ocp-async-1,59d18b526c40aeba,9ff909f912d99cd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:53.255 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:53.255 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:54.355 INFO 26087 --- [prometheus-minute-schedule-1,26d56ee3e5c86cca,88a6af0b31b685e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:54.588 WARN 26087 --- [ocp-async-5,3d5816c15aca5c7e,748abed1a382d5d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:54.588 INFO 26087 --- [ocp-async-8,34801cede98a2fad,ad017911d4beb778] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:56.255 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:56.256 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:12:56.526 INFO 26087 --- [ocp-schedules-29,78e7c36bc153fc39,66c49431940b47af] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:12:56.526 INFO 26087 --- [ocp-schedules-25,8aba0d49d24a75dc,c3ce7ee7a3fdf9a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:12:56.546 INFO 26087 --- [ocp-async-2,414fac09009fc211,bdc16384479deaa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:12:56.588 WARN 26087 --- [ocp-async-3,c189dab7955d50ac,18d1613758673a2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:56.588 INFO 26087 --- [ocp-async-6,5b15e5972baa5ad6,ce24f11f7c7947cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:12:56.623431+08:00, lastProcessTime=2025-04-08T21:12:46.622580+08:00 2025-04-08 21:12:56.625 INFO 26087 --- [ocp-schedules-2,2ab8d22c5f491334,8cc7965aaefae0a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:12:56.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:12:56.623431+08:00 2025-04-08 21:12:56.998 INFO 26087 --- [ocp-schedules-27,22c4c2d3630a62be,a81fe9d0c20692af] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:12:57.017 INFO 26087 --- [ocp-schedules-27,22c4c2d3630a62be,a81fe9d0c20692af] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:12:57.017 INFO 26087 --- [ocp-schedules-27,22c4c2d3630a62be,a81fe9d0c20692af] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:12:57.280 INFO 26087 --- [ocp-schedules-1,6aa964d6b67983f4,83768a0d7e79bbd3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:12:57.281 INFO 26087 --- [ocp-schedules-1,6aa964d6b67983f4,83768a0d7e79bbd3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:12:57.355 INFO 26087 --- [prometheus-minute-schedule-1,7ba8d548dd013ac6,d50ef2f3c27df863] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744117980, interval=60 2025-04-08 21:12:58.589 WARN 26087 --- [ocp-async-10,2177a4656c7fa763,20cd1c2e805ecb4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:12:58.589 INFO 26087 --- [ocp-async-1,aa2538d6adbd4f7e,1ef9a658a9b0e3ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:12:58.734 INFO 26087 --- [ocp-schedules-3,c4945ec3b4fa9329,aafd4c0781cd2d6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:12:59.256 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:12:59.257 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:00.355 INFO 26087 --- [prometheus-minute-schedule-1,3c1ef471c1db2985,d7d9173f796e619f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:00.589 INFO 26087 --- [ocp-async-4,0ed4f0ad54982237,d4143c03714cc38b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:00.589 WARN 26087 --- [ocp-async-8,970d0b6152781069,ccfbee0339442088] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:02.257 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:02.258 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:02.589 INFO 26087 --- [ocp-async-2,837ecc3dba2a1d39,d2cc5819508caf81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:02.589 WARN 26087 --- [ocp-async-9,392f6733018f6d28,f26377ab5a930cbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:03.355 INFO 26087 --- [prometheus-minute-schedule-1,38d579d68670cfe6,fc40afd030fe7965] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:04.589 INFO 26087 --- [ocp-async-6,71c191a5d0a38d31,fb8768de61b016ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:04.589 WARN 26087 --- [ocp-async-7,7d5c0282379bc254,1f69c43e9f101a1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:05.258 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:05.258 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:06.355 INFO 26087 --- [prometheus-minute-schedule-1,82eb6d0068f49ecc,61fab29e315c1401] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:06.355 INFO 26087 --- [prometheus-cache-monitor1,33f6f5d40e51e4f9,a7f81d789786ea11] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1494,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:13:06.355 INFO 26087 --- [prometheus-cache-monitor1,33f6f5d40e51e4f9,a7f81d789786ea11] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:13:06.547 INFO 26087 --- [ocp-async-6,a47fe738d79d5b8e,407f9adc191c9b1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:13:06.589 INFO 26087 --- [ocp-async-8,8077a4ca1c7e89d7,9774542b8683e20b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:06.589 WARN 26087 --- [ocp-async-3,50ff7e00a74742df,94007368a1cf3a88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:13:06.623074+08:00, lastProcessTime=2025-04-08T21:12:56.623431+08:00 2025-04-08 21:13:06.626 INFO 26087 --- [ocp-schedules-10,148cac80de8fea88,61c90cbc978c6d25] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:13:06.759 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:13:06.623074+08:00 2025-04-08 21:13:07.293 INFO 26087 --- [ocp-schedules-6,d4be4167f0c866c8,d3168a33751ba9b4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:13:07.294 INFO 26087 --- [ocp-schedules-6,d4be4167f0c866c8,d3168a33751ba9b4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:13:08.258 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:08.259 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:08.589 INFO 26087 --- [ocp-async-9,c73f353dbf66f7d6,40c308e286ff25c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:08.589 WARN 26087 --- [ocp-async-4,886ee46d1e067cbe,0e0b8a63f655ee26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:08.765 INFO 26087 --- [ocp-schedules-7,29d388ff12bf6c9f,99d81558a7dd198d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:13:09.355 INFO 26087 --- [prometheus-minute-schedule-1,cbc2331da7209049,d3649cdbf2aca93e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:10.001 INFO 26087 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:13:10.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:13:10.589 INFO 26087 --- [ocp-async-7,12dcf9f5eea19409,567db26a4dbba577] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:10.589 WARN 26087 --- [ocp-async-10,b26960c2498c3b58,e8b34734d19d7178] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:10.604 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:13:11.259 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:11.260 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:12.355 INFO 26087 --- [prometheus-minute-schedule-1,3d4761c353b6344c,d4b66ed75f064dc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:12.376 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117920, nodeCount=7, affectRows=7 2025-04-08 21:13:12.377 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117920, nodeCount=6, validCount=6 2025-04-08 21:13:12.390 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117920, nodeCount=10, affectRows=10 2025-04-08 21:13:12.390 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117920, nodeCount=17, affectRows=17 2025-04-08 21:13:12.390 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117920, nodeCount=10, validCount=10 2025-04-08 21:13:12.390 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117920, nodeCount=17, validCount=17 2025-04-08 21:13:12.402 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117920, nodeCount=116, affectRows=116 2025-04-08 21:13:12.402 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117920, nodeCount=106, affectRows=106 2025-04-08 21:13:12.403 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117920, nodeCount=118, affectRows=118 2025-04-08 21:13:12.403 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117920, nodeCount=105, validCount=105 2025-04-08 21:13:12.403 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117920, nodeCount=115, validCount=115 2025-04-08 21:13:12.403 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117920, nodeCount=118, validCount=118 2025-04-08 21:13:12.590 WARN 26087 --- [ocp-async-3,efc5fc8d96e60084,053a97321bb75827] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:12.590 INFO 26087 --- [ocp-async-7,7fccb3775773bd92,b366130db5eb2300] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:14.260 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:14.260 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:14.590 WARN 26087 --- [ocp-async-2,486af47a3fd1b704,a898e8e1b9bdd6e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:14.590 INFO 26087 --- [ocp-async-7,2e33416913112e35,3203e23e27c9b452] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:15.355 INFO 26087 --- [prometheus-minute-schedule-1,db9e3b8d069f9924,b817a1a874668dc3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:16.517 INFO 26087 --- [ocp-schedules-15,da820ea2a1c1f30a,389bc013ebbafebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:13:16.517 INFO 26087 --- [ocp-schedules-24,5df00a67640ee2c7,bea190b0e74c0a6e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:13:16.521 INFO 26087 --- [ocp-schedules-9,288145f3116bec23,5982a797030bbe9c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:13:16.521 INFO 26087 --- [ocp-schedules-14,f98b58cb8d72dfd9,fddc392bf7d21fbd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:13:16.521 INFO 26087 --- [ocp-schedules-14,f98b58cb8d72dfd9,fddc392bf7d21fbd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:13:16.523 INFO 26087 --- [ocp-schedules-25,80fde6d2b8b5d9cd,c690798dd853b138] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:13:16.523 INFO 26087 --- [ocp-schedules-25,80fde6d2b8b5d9cd,c690798dd853b138] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:13:16.524 INFO 26087 --- [ocp-schedules-18,39583ccd1d237e08,b52c2b1489dc6e1c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:13:16.524 INFO 26087 --- [ocp-schedules-18,39583ccd1d237e08,b52c2b1489dc6e1c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:13:16.524 INFO 26087 --- [ocp-schedules-28,885db2feca5e1dd4,3a3c27eb5c375e26] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:13:16.524 INFO 26087 --- [ocp-schedules-5,7d963e6bf0fd765a,f7b48e252055a03c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:13:16.525 INFO 26087 --- [ocp-schedules-24,5df00a67640ee2c7,bea190b0e74c0a6e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:13:16.525 INFO 26087 --- [ocp-schedules-28,885db2feca5e1dd4,3a3c27eb5c375e26] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.525 INFO 26087 --- [ocp-schedules-5,7d963e6bf0fd765a,f7b48e252055a03c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:13:16.535 INFO 26087 --- [ocp-schedules-29,0077a11b343189aa,9062b48858a80415] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:13:16.535 INFO 26087 --- [ocp-schedules-29,0077a11b343189aa,9062b48858a80415] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:13:16.542 INFO 26087 --- [ocp-schedules-15,da820ea2a1c1f30a,389bc013ebbafebb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:13:16.547 INFO 26087 --- [ocp-async-9,ebe01dd24854a9e0,f2791e9d8f58501c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:13:16.550 INFO 26087 --- [ocp-schedules-17,b4f7ca1c4e8b7007,d26b0a3a9f48a374] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:13:16.553 INFO 26087 --- [pool-overview-container-7,406c45eb992ad96d,989095a34ea2ac0e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:13:16.553 INFO 26087 --- [pool-overview-container-8,ee15062fd4d4fdf9,f1a6b0b2e028c2d1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:13:16.553 INFO 26087 --- [pool-overview-container-5,f6045ab58386d6e4,28aa7cbf6a5818ad] o.o.o.i.r.ClusterResourceStatServiceImpl : Cluster sgkydb is not connectable, skip to collect stats 2025-04-08 21:13:16.558 INFO 26087 --- [ocp-schedules-3,62f8f73e9cf9d29f,0bf117d9fd9d1e37] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:13:16.558 INFO 26087 --- [ocp-schedules-3,62f8f73e9cf9d29f,0bf117d9fd9d1e37] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:13:16.560 INFO 26087 --- [pool-11-thread-11,406c45eb992ad96d,c27a68ecec081f4b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:13:16.560 INFO 26087 --- [pool-11-thread-10,406c45eb992ad96d,219337743b41638c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:13:16.561 INFO 26087 --- [pool-11-thread-12,406c45eb992ad96d,ede1fcdfdab356dd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:13:16.562 INFO 26087 --- [ocp-schedules-6,7576b57461990473,8b996bc7e3048922] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:13:16.562 INFO 26087 --- [pool-11-thread-9,406c45eb992ad96d,b8807e7a97b6e0b1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:13:16.562 INFO 26087 --- [ForkJoinPool-7-worker-13,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:13:16.563 INFO 26087 --- [pool-11-thread-10,406c45eb992ad96d,219337743b41638c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:13:16.563 INFO 26087 --- [pool-11-thread-10,406c45eb992ad96d,219337743b41638c] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:13:16.563 INFO 26087 --- [ocp-schedules-6,7576b57461990473,8b996bc7e3048922] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.563 INFO 26087 --- [ForkJoinPool-7-worker-13,406c45eb992ad96d,0d7f9f194dc48f08] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of sgkydb 2025-04-08 21:13:16.563 INFO 26087 --- [ForkJoinPool-7-worker-13,406c45eb992ad96d,0d7f9f194dc48f08] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000002, name=sgkydb, obClusterId=1744016068, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:13:16.563 INFO 26087 --- [ForkJoinPool-7-worker-13,406c45eb992ad96d,0d7f9f194dc48f08] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of sgkydb 2025-04-08 21:13:16.563 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:13:16.564 INFO 26087 --- [pool-11-thread-11,406c45eb992ad96d,c27a68ecec081f4b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:13:16.572 INFO 26087 --- [ocp-schedules-16,501ae53efe84cfc3,b91330ecb26953f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:13:16.573 INFO 26087 --- [ocp-schedules-16,e0c36e1ef4c03027,74fb59af1fd1020a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:13:16.573 INFO 26087 --- [ocp-schedules-16,e0c36e1ef4c03027,74fb59af1fd1020a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.579 INFO 26087 --- [pool-11-thread-9,406c45eb992ad96d,b8807e7a97b6e0b1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:13:16.580 INFO 26087 --- [ForkJoinPool-7-worker-13,406c45eb992ad96d,0d7f9f194dc48f08] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:13:16.580 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:13:16.580 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:13:16.581 INFO 26087 --- [ForkJoinPool-7-worker-13,406c45eb992ad96d,ec0db21d00403060] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000002, name=sgkydb, obClusterId=1744016068, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:13:16.581 INFO 26087 --- [pool-11-thread-12,406c45eb992ad96d,ede1fcdfdab356dd] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:13:16.581 INFO 26087 --- [pool-11-thread-9,406c45eb992ad96d,b8807e7a97b6e0b1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:13:16.582 INFO 26087 --- [pool-11-thread-9,406c45eb992ad96d,b8807e7a97b6e0b1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:13:16.583 INFO 26087 --- [ocp-schedules-9,d80ab1c2353f8f98,04847a32d77abb0f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:13:16.585 INFO 26087 --- [ocp-schedules-18,fa56d080964d3de2,59638b937754f8ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1494,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":372,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:13:16.585 INFO 26087 --- [ocp-schedules-18,fa56d080964d3de2,59638b937754f8ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:13:16.590 INFO 26087 --- [ocp-async-6,3e22a6709ff247b7,bd6f4fe869ccb21b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:16.590 WARN 26087 --- [ocp-async-5,4a0efabc26531293,28ff581ef45d4abc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:16.592 INFO 26087 --- [ocp-schedules-8,721a69ebd2f80012,807ce5ddee2d9c51] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:13:16.593 INFO 26087 --- [ocp-schedules-8,721a69ebd2f80012,807ce5ddee2d9c51] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.594 INFO 26087 --- [ocp-schedules-12,80d969dc628a07b8,3e2bc09c6593d703] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 21:13:16.594 INFO 26087 --- [ocp-schedules-12,80d969dc628a07b8,3e2bc09c6593d703] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 21:13:16.594 INFO 26087 --- [ocp-schedules-12,80d969dc628a07b8,3e2bc09c6593d703] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=426 2025-04-08 21:13:16.595 INFO 26087 --- [ocp-schedules-19,76d0eaa12f1c9dad,5aeaf34130ccb597] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:13:16.609 INFO 26087 --- [ocp-schedules-30,82e8026b95869aac,dc9d778ccf3af51a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:13:16.609 INFO 26087 --- [ocp-schedules-30,82e8026b95869aac,dc9d778ccf3af51a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:13:16.617 INFO 26087 --- [ocp-schedules-11,4fb011bdf1a66941,f5332401d08d1a2e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:13:16.617 INFO 26087 --- [ocp-schedules-10,acfacd4548085294,df45c7f5b6a8db1a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:13:16.617+08:00, clusters=[1000002] 2025-04-08 21:13:16.617 INFO 26087 --- [ocp-schedules-12,80d969dc628a07b8,3e2bc09c6593d703] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=426 2025-04-08 21:13:16.618 INFO 26087 --- [ocp-schedules-10,acfacd4548085294,df45c7f5b6a8db1a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:13:16.619 INFO 26087 --- [ocp-schedules-13,db3af20d941870f6,ebb7af20a648f5ef] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:13:16.619+08:00, clusters=[1000002] 2025-04-08 21:13:16.622 INFO 26087 --- [ocp-schedules-13,db3af20d941870f6,ebb7af20a648f5ef] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:13:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:13:16.623517+08:00, lastProcessTime=2025-04-08T21:13:06.623074+08:00 2025-04-08 21:13:16.627 INFO 26087 --- [ocp-schedules-26,62561b140918e066,76b839ca788afce1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:13:16.631 INFO 26087 --- [ocp-schedules-3,0c122633b8b945be,08c42b961862e0d2] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:13:16.631 INFO 26087 --- [ocp-schedules-3,0c122633b8b945be,08c42b961862e0d2] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:13:16.638 INFO 26087 --- [ocp-schedules-4,25dcc475ff2294ed,a6a25a92c42f6409] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:13:16.638 INFO 26087 --- [ocp-schedules-4,25dcc475ff2294ed,a6a25a92c42f6409] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:13:16.645 INFO 26087 --- [ocp-schedules-4,25dcc475ff2294ed,a6a25a92c42f6409] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:13:16.645 INFO 26087 --- [ocp-schedules-4,25dcc475ff2294ed,a6a25a92c42f6409] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:13:16.661 INFO 26087 --- [ocp-schedules-22,59cb7b28946fa573,605a86ae05b769ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:13:16.663 INFO 26087 --- [ocp-schedules-22,59cb7b28946fa573,605a86ae05b769ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:13:16.663 INFO 26087 --- [ocp-schedules-22,59cb7b28946fa573,605a86ae05b769ee] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:13:16.714 INFO 26087 --- [ocp-schedules-2,af1648de95e8d398,5fdd31df7fb0943f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:13:16.717 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:13:16.623517+08:00 2025-04-08 21:13:16.748 INFO 26087 --- [ocp-schedules-14,cfb6eb5b9cba48c3,14cea6b31fdc9557] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:13:16.749 INFO 26087 --- [ocp-schedules-14,cfb6eb5b9cba48c3,14cea6b31fdc9557] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.760 INFO 26087 --- [ocp-schedules-25,b6702e6ad9a9bd56,cea9a869b2af4d8b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 21:13:16.761 INFO 26087 --- [ocp-schedules-25,b6702e6ad9a9bd56,cea9a869b2af4d8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.768 INFO 26087 --- [ocp-schedules-21,3b78ed4e1e0035a1,113fbc7a8a3286a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:13:16.770 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:13:16.771 ERROR 26087 --- [ocp-schedules-14,cfb6eb5b9cba48c3,14cea6b31fdc9557] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:13:16.771 INFO 26087 --- [ocp-schedules-14,cfb6eb5b9cba48c3,14cea6b31fdc9557] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:13:16.772 ERROR 26087 --- [ocp-schedules-25,b6702e6ad9a9bd56,cea9a869b2af4d8b] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:13:16.772 INFO 26087 --- [ocp-schedules-25,b6702e6ad9a9bd56,cea9a869b2af4d8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 21:13:16.797 INFO 26087 --- [ocp-schedules-9,b976ccd196b4ed56,712797f493002329] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:13:16.798 INFO 26087 --- [ocp-schedules-9,b976ccd196b4ed56,712797f493002329] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:16.807 INFO 26087 --- [ocp-schedules-29,5a2a3408e74a3db9,73cfd28d3dd118e3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:13:16.807 INFO 26087 --- [ocp-schedules-29,5a2a3408e74a3db9,73cfd28d3dd118e3] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:13:16.812 ERROR 26087 --- [ocp-schedules-9,b976ccd196b4ed56,712797f493002329] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:13:16.812 INFO 26087 --- [ocp-schedules-9,b976ccd196b4ed56,712797f493002329] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:13:16.815 INFO 26087 --- [pool-26-thread-1,5a2a3408e74a3db9,adeae153d47e4a72] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:13:16.818 INFO 26087 --- [ocp-schedules-24,5a80e2674213efa9,8ce1e7e84e370a9e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:13:16.818 INFO 26087 --- [ocp-schedules-24,5a80e2674213efa9,8ce1e7e84e370a9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:13:16.821 INFO 26087 --- [ocp-schedules-24,5a80e2674213efa9,8ce1e7e84e370a9e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:13:16.822 INFO 26087 --- [ocp-schedules-24,5a80e2674213efa9,8ce1e7e84e370a9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208338840276992 2025-04-08 21:13:16.822 WARN 26087 --- [ocp-schedules-29,5a2a3408e74a3db9,73cfd28d3dd118e3] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$14662e73.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:13:16.822 INFO 26087 --- [ocp-schedules-29,5a2a3408e74a3db9,73cfd28d3dd118e3] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:13:16.822 INFO 26087 --- [ocp-schedules-24,5a80e2674213efa9,8ce1e7e84e370a9e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:13:16.822 INFO 26087 --- [ocp-schedules-24,5a80e2674213efa9,8ce1e7e84e370a9e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:13:17.018 INFO 26087 --- [ocp-schedules-15,73721022c3515744,d00ab254d6d7d675] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:13:17.034 INFO 26087 --- [ocp-schedules-15,73721022c3515744,d00ab254d6d7d675] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:13:17.034 INFO 26087 --- [ocp-schedules-15,73721022c3515744,d00ab254d6d7d675] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:13:17.123 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:13:17.134 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:13:17.136 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:13:17.136 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:13:17.137 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:13:17.139 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:13:17.140 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:13:17.140 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:13:17.141 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:13:17.144 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:13:17.145 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:13:17.146 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:13:17.146 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:13:17.149 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:13:17.150 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:13:17.150 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:13:17.151 WARN 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:13:17.154 INFO 26087 --- [ocp-schedules-17,c5978989fe73a27d,9be24c5caa78e897] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=4, cachedSize=6 2025-04-08 21:13:17.260 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:17.261 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:17.302 INFO 26087 --- [ocp-schedules-11,551cc131c03ca307,824d4ee2964bca1b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:13:17.303 INFO 26087 --- [ocp-schedules-11,551cc131c03ca307,824d4ee2964bca1b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:13:17.478 INFO 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:13:17.478 INFO 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:13:17.478 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:13:17.478 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:13:17.478 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:13:17.481 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:13:17.481 INFO 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:13:17.481 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:13:17.481 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:13:17.482 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:13:17.482 WARN 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:13:17.482 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:13:17.482 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:13:17.482 INFO 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:13:17.482 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:13:17.482 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:13:17.482 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:13:17.483 WARN 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:13:17.483 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:13:17.483 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:13:17.483 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:13:17.486 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:13:17.486 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:13:17.486 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:13:17.487 INFO 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:13:17.489 WARN 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:13:17.492 ERROR 26087 --- [ocp-schedules-3,e2a0aebee9bc6624,47f9d18b650553a7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2025-04-08 21:13:18.355 INFO 26087 --- [prometheus-minute-schedule-1,bad257d56f12d880,5dab9ca109083336] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:18.590 WARN 26087 --- [ocp-async-2,2075cd0dd8d39d1a,5e0c78d74f43bb34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:18.590 INFO 26087 --- [ocp-async-9,21cf367f2d90e24b,0902b0cff674f5e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:18.788 INFO 26087 --- [ocp-schedules-3,d3599880b603e274,e01ecc131a6e97b2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:13:20.261 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:20.262 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:20.590 WARN 26087 --- [ocp-async-9,1348a35c07edc87b,ead628dcf9e3ca50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:20.590 INFO 26087 --- [ocp-async-10,783ff03f17c2427a,b3580285fcc89785] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:13:21.355 INFO 26087 --- [prometheus-minute-schedule-1,6d935ecec44036ff,11f6bcb4f6e11ade] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:22.590 WARN 26087 --- [ocp-async-3,43086b32a89d94b0,2a64a736e2471bb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:22.590 INFO 26087 --- [ocp-async-5,5026c2eb3a78966d,6a49d130cfb83c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:23.262 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:23.263 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:24.355 INFO 26087 --- [prometheus-minute-schedule-1,cd3c4d80416f7a20,e7600fbd2d399d17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:24.590 WARN 26087 --- [ocp-async-5,6a9939ec79763d05,852d07d132de1a68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:24.590 INFO 26087 --- [ocp-async-2,a77f6f9c95353649,c5f3d8207c02c182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:26.263 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:26.263 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:26.525 INFO 26087 --- [ocp-schedules-28,885db2feca5e1dd4,3a3c27eb5c375e26] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:13:26.525 INFO 26087 --- [ocp-schedules-5,7d963e6bf0fd765a,f7b48e252055a03c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:13:26.547 INFO 26087 --- [ocp-async-9,7d891d2abd85d605,b9ca40de7425aa62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:13:26.563 INFO 26087 --- [ocp-schedules-6,7576b57461990473,8b996bc7e3048922] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:13:26.590 WARN 26087 --- [ocp-async-10,dec9491769ea81d4,b9bec74b735aedd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:26.591 INFO 26087 --- [ocp-async-3,cb28bf17539cbe2a,42dfc11677b67608] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:26.594 INFO 26087 --- [ocp-schedules-8,721a69ebd2f80012,807ce5ddee2d9c51] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 21:13:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:13:26.622589+08:00, lastProcessTime=2025-04-08T21:13:16.623517+08:00 2025-04-08 21:13:26.628 INFO 26087 --- [ocp-schedules-30,472745ca4dafbdd1,a16eebd436459067] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:13:26.737 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:13:26.622589+08:00 2025-04-08 21:13:27.334 INFO 26087 --- [ocp-schedules-3,2b05267e9b53fd0d,c7b99843d6c2c08e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:13:27.335 INFO 26087 --- [ocp-schedules-3,2b05267e9b53fd0d,c7b99843d6c2c08e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:13:27.355 INFO 26087 --- [prometheus-minute-schedule-1,f020c33d302fb495,e060288856d6c2c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:28.591 WARN 26087 --- [ocp-async-1,938e032674c7fbaa,eaf4c2adcb80ee9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:28.591 INFO 26087 --- [ocp-async-5,5b6ef4d9a4cb3895,90f2e949345d34d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:28.814 INFO 26087 --- [ocp-schedules-18,64ad43c854046573,20450a63d2083858] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:13:29.263 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:29.264 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:30.355 INFO 26087 --- [prometheus-minute-schedule-1,23e9f01415c098ff,b716c0c2fc5c366a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:30.591 WARN 26087 --- [ocp-async-5,073a2f183f1ec198,fd5fa786501195c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:30.591 INFO 26087 --- [ocp-async-8,5d6b322b657b4668,810b64cb6f624024] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:32.264 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:32.265 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:32.591 WARN 26087 --- [ocp-async-7,13ffe5a289b323f3,d386bcd0cd878443] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:32.591 INFO 26087 --- [ocp-async-6,51480317eb82ce9e,c26ff09acd42dea4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:33.355 INFO 26087 --- [prometheus-minute-schedule-1,b0f18472173f3eb4,1fbdecc86122f98d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:34.591 WARN 26087 --- [ocp-async-4,097c44bdec9f7439,1c9443ced7ea2793] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:34.591 INFO 26087 --- [ocp-async-1,fd377a835de7cc4a,6c8f897748e12f3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:34.978 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:13:34.979 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:13:34.979 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:13:34.980 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:13:34.980 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:13:34.986 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster sgkydb(1000002), status=TAKINGOVER 2025-04-08 21:13:34.986 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=6ms, period=120000ms 2025-04-08 21:13:35.265 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:35.266 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:36.355 INFO 26087 --- [prometheus-minute-schedule-1,26d329523255f0a6,12b99733ab96f6c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:36.547 INFO 26087 --- [ocp-async-1,f1ec31ae94e838fc,3a90f85d69d74fd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:13:36.591 WARN 26087 --- [ocp-async-2,7b965221c31b5dfe,52104394fbd5a791] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:36.591 INFO 26087 --- [ocp-async-10,0529b04991ec81fb,7232ad4b5f4aec45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:13:36.622698+08:00, lastProcessTime=2025-04-08T21:13:26.622589+08:00 2025-04-08 21:13:36.629 INFO 26087 --- [ocp-schedules-11,8a11ab30076e28b6,d6cd12326ee1b84a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:13:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:13:36.622698+08:00 2025-04-08 21:13:37.036 INFO 26087 --- [ocp-schedules-9,0b74f201cb0123b8,960af98653fc1d52] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:13:37.090 INFO 26087 --- [ocp-schedules-9,0b74f201cb0123b8,960af98653fc1d52] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:13:37.090 INFO 26087 --- [ocp-schedules-9,0b74f201cb0123b8,960af98653fc1d52] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:13:37.355 INFO 26087 --- [ocp-schedules-12,5de15acf1671a17d,9091d429f34f74ba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:13:37.356 INFO 26087 --- [ocp-schedules-12,5de15acf1671a17d,9091d429f34f74ba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:13:38.266 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:38.266 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:38.591 WARN 26087 --- [ocp-async-7,64b601e467761a8e,087cb494804e0851] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:38.591 INFO 26087 --- [ocp-async-8,d01cbd71fe7f5d7c,988f31f156dd6316] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:38.836 INFO 26087 --- [ocp-schedules-27,32ef4f9f97560d06,666b2f49962f814d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:13:39.355 INFO 26087 --- [prometheus-minute-schedule-1,267f7093a258bb81,21c09f5de59d0e9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:40.001 INFO 26087 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:13:40.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:13:40.591 WARN 26087 --- [ocp-async-4,ceb1be84d28d9b46,bb700c9e6779ae14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:40.591 INFO 26087 --- [ocp-async-9,36d122967547770c,83d94b51c6208879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:41.266 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:41.267 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:42.355 INFO 26087 --- [prometheus-minute-schedule-1,dcb637a94bac46c1,61cd022a090ce495] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:42.592 WARN 26087 --- [ocp-async-2,c927ac8121f05fb9,48109ebffeb5e703] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:42.592 INFO 26087 --- [ocp-async-9,306fb726dbac2a6b,3b0055c887df52e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:44.267 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:44.268 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:44.592 WARN 26087 --- [ocp-async-9,037f29dc1313a478,5f38808886bbbf55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:44.592 INFO 26087 --- [ocp-async-6,8863fc0b9f8e8699,0ba02d51ad3c7d2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:45.355 INFO 26087 --- [prometheus-minute-schedule-1,f9c579c68e6de3e3,eda5c75cd89ad466] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:46.521 INFO 26087 --- [ocp-schedules-8,b9d26e9b50349f7d,d1c81d2f26d4199c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:13:46.521 INFO 26087 --- [ocp-schedules-27,b0b3ccbc3d70099e,29ea6bad91db4d87] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:13:46.521 INFO 26087 --- [ocp-schedules-27,b0b3ccbc3d70099e,29ea6bad91db4d87] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:13:46.523 INFO 26087 --- [ocp-schedules-14,69781303d570e4b6,a2434da2c1d493ba] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:13:46.523 INFO 26087 --- [ocp-schedules-14,69781303d570e4b6,a2434da2c1d493ba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:13:46.524 INFO 26087 --- [ocp-schedules-25,24f6e435e2773482,662d3663633a6d3e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:13:46.524 INFO 26087 --- [ocp-schedules-25,24f6e435e2773482,662d3663633a6d3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:13:46.524 INFO 26087 --- [ocp-schedules-12,a9d1402ef69b37fc,7334879685304d27] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:13:46.524 INFO 26087 --- [ocp-schedules-28,9988cb9c61c42c77,a0a04badb09595f7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:13:46.524 INFO 26087 --- [ocp-schedules-3,c8fc6e7be2fb7103,1a29a828c0290055] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:13:46.525 INFO 26087 --- [ocp-schedules-3,c8fc6e7be2fb7103,1a29a828c0290055] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:46.525 INFO 26087 --- [ocp-schedules-28,9988cb9c61c42c77,a0a04badb09595f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:13:46.547 INFO 26087 --- [ocp-async-8,37920d0f49c98052,239652b313585565] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:13:46.581 INFO 26087 --- [ocp-schedules-16,e0c36e1ef4c03027,74fb59af1fd1020a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:13:46.581 INFO 26087 --- [ocp-schedules-16,4efba6b3a68f15ed,0f41670222424fa2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:13:46.582 INFO 26087 --- [ocp-schedules-16,4efba6b3a68f15ed,0f41670222424fa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:13:46.589 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:13:46.589 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:13:46.592 WARN 26087 --- [ocp-async-4,4452d0ce7e15b0c5,c1c53dcc74bc9b71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:46.592 INFO 26087 --- [ocp-async-3,30a595605ae47eee,aacf4d3fdb148733] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:46.596 INFO 26087 --- [ocp-schedules-15,d5ec1d5cb8903e90,4e3dc0edd06dd783] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:13:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:13:46.623288+08:00, lastProcessTime=2025-04-08T21:13:36.622698+08:00 2025-04-08 21:13:46.629 INFO 26087 --- [ocp-schedules-29,95e6441d078181cb,396ee73648cc0d30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:13:46.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:13:46.623288+08:00 2025-04-08 21:13:47.268 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:47.268 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:47.361 INFO 26087 --- [ocp-schedules-25,1d07551e4e20855c,07fa87447e1c93f2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:13:47.362 INFO 26087 --- [ocp-schedules-25,1d07551e4e20855c,07fa87447e1c93f2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:13:48.355 INFO 26087 --- [prometheus-minute-schedule-1,7d88d4099b4a10b4,1627d7dd8fa071de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:48.592 WARN 26087 --- [ocp-async-2,e6c758b180958102,d8103c5fa7da23c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:48.592 INFO 26087 --- [ocp-async-5,248dba16cef09774,d324cd71067f305e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:48.864 INFO 26087 --- [ocp-schedules-25,e53548f1edf35406,c415bed6b8b4cd36] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:13:50.268 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:50.269 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:50.592 WARN 26087 --- [ocp-async-6,c80309bb60a6e2c4,6c443c279c28b94d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:50.592 INFO 26087 --- [ocp-async-9,50415292f6f8327b,52ddc03b0f81d5ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:51.355 INFO 26087 --- [prometheus-minute-schedule-1,2fa6dbc28a8f87c1,82a281943f38762d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:52.592 WARN 26087 --- [ocp-async-4,393dea5851d7c9c7,0953c57832723c5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:52.592 INFO 26087 --- [ocp-async-4,652594f7825f1d7e,b02b6012924f3da4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:53.269 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:53.270 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:54.355 INFO 26087 --- [prometheus-minute-schedule-1,c4f2ae9e174ffdc1,de7df74fdd156891] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:54.592 WARN 26087 --- [ocp-async-2,e320483496fa6d9b,dcd08a7f233bb655] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:54.592 INFO 26087 --- [ocp-async-10,60644777aad2345c,e87841fa48d387be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:56.270 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:56.270 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:13:56.525 INFO 26087 --- [ocp-schedules-3,c8fc6e7be2fb7103,1a29a828c0290055] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:13:56.526 INFO 26087 --- [ocp-schedules-28,9988cb9c61c42c77,a0a04badb09595f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:13:56.547 INFO 26087 --- [ocp-async-10,1d81354bd9d370ff,7e06aff772dc7d9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:13:56.593 WARN 26087 --- [ocp-async-6,c6e653ae379aef2b,9fccc5967f686b69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:56.593 INFO 26087 --- [ocp-async-9,d28a775daec7fafe,ed68adc9f32d1ec0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:13:56.623713+08:00, lastProcessTime=2025-04-08T21:13:46.623288+08:00 2025-04-08 21:13:56.631 INFO 26087 --- [ocp-schedules-15,d5e2808290d22ffd,5c91aefb0e1f8fac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:13:56.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:13:56.623713+08:00 2025-04-08 21:13:57.091 INFO 26087 --- [ocp-schedules-4,befb05308efb4b7c,f2be066184c2d945] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:13:57.101 INFO 26087 --- [ocp-schedules-4,befb05308efb4b7c,f2be066184c2d945] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:13:57.101 INFO 26087 --- [ocp-schedules-4,befb05308efb4b7c,f2be066184c2d945] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:13:57.355 INFO 26087 --- [prometheus-minute-schedule-1,cc023b04840cc40f,3d18f81e6f6cfcc2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118040, interval=60 2025-04-08 21:13:57.365 INFO 26087 --- [ocp-schedules-7,dfeb4f4cf16b38e0,811a4ff149cf959d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:13:57.366 INFO 26087 --- [ocp-schedules-7,dfeb4f4cf16b38e0,811a4ff149cf959d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:13:58.593 WARN 26087 --- [ocp-async-1,c0ce3fe4d9c92066,268340c342952c7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:13:58.593 INFO 26087 --- [ocp-async-4,7b7c2bdd024d873c,bd452fd13234f423] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:13:58.884 INFO 26087 --- [ocp-schedules-7,b2a076e34a111637,210ab3f9ba9f1027] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:13:59.270 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:13:59.271 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:00.355 INFO 26087 --- [prometheus-minute-schedule-1,3569f1bcb5a683d7,0aeef4bbca6af825] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:00.593 WARN 26087 --- [ocp-async-3,0ec91c9482024665,17a89408a8633954] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:00.593 INFO 26087 --- [ocp-async-2,d70d8ec63b10f4a8,85001fae2d224c53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:02.271 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:02.272 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:02.593 INFO 26087 --- [ocp-async-6,967c07f384081850,1fdaf2d0259fc2a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:02.593 WARN 26087 --- [ocp-async-10,c5acd379e35f51c4,77e518d012073d8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:03.355 INFO 26087 --- [prometheus-minute-schedule-1,1d94ef03e60c8d24,45fd0f655773711d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:04.593 WARN 26087 --- [ocp-async-1,a84ac03c8eb9c9ae,389440ce77ecb0fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:04.593 INFO 26087 --- [ocp-async-8,fa55d0b482bab545,73a1ba18155e37ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:05.272 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:05.272 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:06.243 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:08:06.202, end=2025-04-08T21:13:06.217} 2025-04-08 21:14:06.247 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:14:06.247 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:14:06.275 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:08:06.206, end=2025-04-08T21:13:06.228} 2025-04-08 21:14:06.277 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:14:06.277 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:14:06.355 INFO 26087 --- [prometheus-minute-schedule-1,fbaf7663b840bb70,5074b52e5f3fe3e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:06.355 INFO 26087 --- [prometheus-cache-monitor1,5439caf14f77146f,6daa287b181e6274] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1494,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:14:06.355 INFO 26087 --- [prometheus-cache-monitor1,5439caf14f77146f,6daa287b181e6274] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:14:06.363 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:10:06.344, end=2025-04-08T21:12:06.36} 2025-04-08 21:14:06.366 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:14:06.366 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:14:06.547 INFO 26087 --- [ocp-async-8,019073dc76272531,b0b5e9602e956bdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:14:06.593 INFO 26087 --- [ocp-async-2,dde3293d85ce45f1,94e2de06dc61c2fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:06.593 WARN 26087 --- [ocp-async-5,9139ce763d6e5d6f,995991a12bca2e56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:14:06.622645+08:00, lastProcessTime=2025-04-08T21:13:56.623713+08:00 2025-04-08 21:14:06.631 INFO 26087 --- [ocp-schedules-29,d456c4853a337a3f,fc14fde98ecf22b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:14:06.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:14:06.622645+08:00 2025-04-08 21:14:07.466 INFO 26087 --- [ocp-schedules-21,63f9e4f9d445604b,7a6df450875e944b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:14:07.467 INFO 26087 --- [ocp-schedules-21,63f9e4f9d445604b,7a6df450875e944b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:14:08.273 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:08.273 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:08.593 WARN 26087 --- [ocp-async-6,65097284f0ef78c9,71ee033b97d58dac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:08.593 INFO 26087 --- [ocp-async-7,413a85c7d5aa8ac0,fd8cd44e4c76cbdb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:08.930 INFO 26087 --- [ocp-schedules-7,c15d638b8ee47715,46eaaa2392374bef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:14:09.355 INFO 26087 --- [prometheus-minute-schedule-1,f3bd09fe26e4c0b1,57fecc572517c027] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:10.002 INFO 26087 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:14:10.002 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:14:10.593 WARN 26087 --- [ocp-async-8,d37fa27efa56a9fc,4e54c7c40ab27190] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:10.593 INFO 26087 --- [ocp-async-1,f3571ff48b1d725b,203422c0cf24ab48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:10.600 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:14:11.273 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:11.274 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:12.355 INFO 26087 --- [prometheus-minute-schedule-1,7eaf1cbbae544b5b,e78580f1dbd0329e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:12.383 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117980, nodeCount=86, affectRows=86 2025-04-08 21:14:12.383 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117980, nodeCount=84, validCount=84 2025-04-08 21:14:12.384 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117980, nodeCount=147, affectRows=147 2025-04-08 21:14:12.385 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744117980, nodeCount=141, affectRows=141 2025-04-08 21:14:12.385 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117980, nodeCount=147, validCount=147 2025-04-08 21:14:12.385 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744117980, nodeCount=140, validCount=140 2025-04-08 21:14:12.594 WARN 26087 --- [ocp-async-4,2d97436b3a743f3e,3a0a64ae5002b121] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:12.594 INFO 26087 --- [ocp-async-1,2dcc39a38748bebd,3ae9d0fee4e601da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:14.274 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:14.274 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:14.594 WARN 26087 --- [ocp-async-6,48a599655bdbd65f,28a56fefb3c7b2a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:14.594 INFO 26087 --- [ocp-async-1,ce8875bc45a05912,119cd4509242da36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:15.355 INFO 26087 --- [prometheus-minute-schedule-1,5b46d7a3a06a5e5e,274701d1b331ffbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:16.517 INFO 26087 --- [ocp-schedules-15,eb19b557a2a4e39f,2d7e1a8d2b6225d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:14:16.517 INFO 26087 --- [ocp-schedules-25,e5ab338ed05661b9,91888d59e5841d28] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:14:16.521 INFO 26087 --- [ocp-schedules-8,ed367b43279ce938,422d919040330f3f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:14:16.522 INFO 26087 --- [ocp-schedules-22,5e065240dac40f8f,707e71ea6d650c21] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:14:16.522 INFO 26087 --- [ocp-schedules-22,5e065240dac40f8f,707e71ea6d650c21] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:14:16.523 INFO 26087 --- [ocp-schedules-20,b45b071ffffbcc9c,d49f670a87191a20] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:14:16.523 INFO 26087 --- [ocp-schedules-20,b45b071ffffbcc9c,d49f670a87191a20] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:14:16.524 INFO 26087 --- [ocp-schedules-2,de1dda6b2e8eb380,732ffea48d9c7116] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:14:16.524 INFO 26087 --- [ocp-schedules-2,de1dda6b2e8eb380,732ffea48d9c7116] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:14:16.524 INFO 26087 --- [ocp-schedules-25,e5ab338ed05661b9,91888d59e5841d28] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:14:16.524 INFO 26087 --- [ocp-schedules-9,557e2668ba73f5e1,f5d1eabebe25a6ab] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:14:16.524 INFO 26087 --- [ocp-schedules-14,f20ae956e47825f0,5ed7f03852bb441a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:14:16.525 INFO 26087 --- [ocp-schedules-9,557e2668ba73f5e1,f5d1eabebe25a6ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:16.525 INFO 26087 --- [ocp-schedules-14,f20ae956e47825f0,5ed7f03852bb441a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:14:16.530 INFO 26087 --- [ocp-schedules-15,eb19b557a2a4e39f,2d7e1a8d2b6225d3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:14:16.534 INFO 26087 --- [ocp-schedules-10,5b1de2ada77455dd,08662ccb22b3c95c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:14:16.548 INFO 26087 --- [ocp-async-8,d08c7dbfc20a0986,eb7371bbb94e0f1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:14:16.559 INFO 26087 --- [ocp-schedules-3,65313fb9226789a6,31bfad8f38e0f7aa] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 21:14:16.559 INFO 26087 --- [ocp-schedules-3,65313fb9226789a6,31bfad8f38e0f7aa] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 21:14:16.572 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:14:16.572 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:14:16.576 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:14:16.576 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:14:16.576 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:14:16.577 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:14:16.577 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:14:16.577 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:14:16.580 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:14:16.580 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:14:16.580 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:14:16.582 INFO 26087 --- [ocp-schedules-30,3848b595e9a46260,061e2d9f6dcd89bc] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:14:16.585 INFO 26087 --- [ocp-schedules-21,fdbe8e683edd3d84,1a504617bd86969f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:14:16.589 INFO 26087 --- [ocp-schedules-16,4efba6b3a68f15ed,0f41670222424fa2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:14:16.589 INFO 26087 --- [ocp-schedules-16,df6b28cd80054c93,3e50561f08e0923e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:14:16.590 INFO 26087 --- [ocp-schedules-16,df6b28cd80054c93,3e50561f08e0923e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:16.594 WARN 26087 --- [ocp-async-9,af9a87ea0f7c5742,18bbcb7616c61e49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:16.594 INFO 26087 --- [ocp-async-3,a5b837ab7a13097d,e9812e3f65139241] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:16.596 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:14:16.596 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:14:16.597 INFO 26087 --- [ocp-schedules-5,bfba366e903e3f27,d74606732cbe2bc6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:14:16.611 INFO 26087 --- [ocp-schedules-2,f25342418a5db581,5242c0d31a5c7b67] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:14:16.611 INFO 26087 --- [ocp-schedules-2,f25342418a5db581,5242c0d31a5c7b67] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:14:16.623 INFO 26087 --- [ocp-schedules-10,9e7c0ace41abfc76,505be37bf5ee269a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:14:16.623 INFO 26087 --- [ocp-schedules-15,12373d71b19ba52a,b60f81a14722c1fb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:14:16.623+08:00, clusters=[1000002] 2025-04-08 21:14:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:14:16.623089+08:00, lastProcessTime=2025-04-08T21:14:06.622645+08:00 2025-04-08 21:14:16.624 INFO 26087 --- [ocp-schedules-15,12373d71b19ba52a,b60f81a14722c1fb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:14:16.627 INFO 26087 --- [ocp-schedules-24,914bf938d835bc26,52ea6156f3dcc4c0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:14:16.627+08:00, clusters=[1000002] 2025-04-08 21:14:16.629 INFO 26087 --- [ocp-schedules-24,914bf938d835bc26,52ea6156f3dcc4c0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:14:16.633 INFO 26087 --- [ocp-schedules-26,a1e0122c2423a3c9,cb647e95e5736d4d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:14:16.633 INFO 26087 --- [ocp-schedules-1,a59f8581dc3a29c8,dda9fedfb22d2028] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:14:16.633 INFO 26087 --- [ocp-schedules-1,a59f8581dc3a29c8,dda9fedfb22d2028] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:14:16.645 INFO 26087 --- [ocp-schedules-17,37e428ad6c5fed5e,f2ad1d1048467278] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:14:16.645 INFO 26087 --- [ocp-schedules-17,37e428ad6c5fed5e,f2ad1d1048467278] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:14:16.651 INFO 26087 --- [ocp-schedules-17,37e428ad6c5fed5e,f2ad1d1048467278] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:14:16.651 INFO 26087 --- [ocp-schedules-17,37e428ad6c5fed5e,f2ad1d1048467278] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:14:16.663 INFO 26087 --- [ocp-schedules-4,55652e480d19e598,fc8af96db388134e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:14:16.665 INFO 26087 --- [ocp-schedules-4,55652e480d19e598,fc8af96db388134e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:14:16.665 INFO 26087 --- [ocp-schedules-4,55652e480d19e598,fc8af96db388134e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:14:16.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:14:16.623089+08:00 2025-04-08 21:14:16.719 INFO 26087 --- [ocp-schedules-18,af886dcded463fcd,20b8a0958bd7e8ac] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:14:16.769 INFO 26087 --- [ocp-schedules-23,b6e3ed721b12e1f3,07033e0285ba9d6d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:14:16.769 INFO 26087 --- [ocp-schedules-23,b6e3ed721b12e1f3,07033e0285ba9d6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:16.771 INFO 26087 --- [ocp-schedules-29,824a85662499437d,716b75f143826c12] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:14:16.772 INFO 26087 --- [ocp-schedules-29,824a85662499437d,716b75f143826c12] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:16.783 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:14:16.784 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:14:16.791 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:14:16.792 ERROR 26087 --- [ocp-schedules-29,824a85662499437d,716b75f143826c12] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:14:16.792 INFO 26087 --- [ocp-schedules-29,824a85662499437d,716b75f143826c12] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:14:16.825 INFO 26087 --- [ocp-schedules-19,453b23f0388a4395,8ba8ea10c930d271] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:14:16.825 INFO 26087 --- [ocp-schedules-19,453b23f0388a4395,8ba8ea10c930d271] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:14:16.825 INFO 26087 --- [ocp-schedules-19,453b23f0388a4395,8ba8ea10c930d271] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:14:16.825 INFO 26087 --- [ocp-schedules-19,453b23f0388a4395,8ba8ea10c930d271] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208338840276992 2025-04-08 21:14:16.825 INFO 26087 --- [ocp-schedules-19,453b23f0388a4395,8ba8ea10c930d271] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:14:16.825 INFO 26087 --- [ocp-schedules-19,453b23f0388a4395,8ba8ea10c930d271] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:14:17.102 INFO 26087 --- [ocp-schedules-6,e4043aa81518ff7c,f8484683a56ca034] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:14:17.107 INFO 26087 --- [ocp-schedules-6,e4043aa81518ff7c,f8484683a56ca034] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:14:17.107 INFO 26087 --- [ocp-schedules-6,e4043aa81518ff7c,f8484683a56ca034] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:14:17.274 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:17.275 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:17.477 INFO 26087 --- [ocp-schedules-2,21032884023e625f,92af8e40455dc1da] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:14:17.479 INFO 26087 --- [ocp-schedules-2,21032884023e625f,92af8e40455dc1da] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:14:17.495 INFO 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:14:17.495 INFO 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:14:17.495 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:14:17.495 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:14:17.495 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:14:17.498 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:14:17.498 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:14:17.498 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:14:17.498 INFO 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:14:17.499 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:14:17.499 WARN 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:14:17.499 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:14:17.499 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:14:17.500 INFO 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:14:17.500 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:14:17.500 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:14:17.500 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:14:17.500 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:14:17.500 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:14:17.501 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:14:17.501 WARN 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:14:17.504 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:14:17.504 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:14:17.504 INFO 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:14:17.505 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:14:17.507 WARN 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:14:17.508 ERROR 26087 --- [ocp-schedules-25,48f2dd7a39e5f647,7824cd3d1e67948c] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 15 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.62:62889 [/192.168.1.62] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 41 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.62:62889 [/192.168.1.62] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 55 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 64 common frames omitted 2025-04-08 21:14:18.355 INFO 26087 --- [prometheus-minute-schedule-1,94986953266c1325,b97aa5a568f0ee44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:18.594 WARN 26087 --- [ocp-async-10,6640890a2ecd150d,6896e012fb4e3d2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:18.594 INFO 26087 --- [ocp-async-3,c20380ec7f19dd45,f6b495bf09c424cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:18.952 INFO 26087 --- [ocp-schedules-8,26bd0a6085627383,9a386d4a9279c342] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:14:20.275 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:20.276 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:14:20.594 WARN 26087 --- [ocp-async-1,b11f9b6de8e15ce0,39849dc96ce77672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:20.594 INFO 26087 --- [ocp-async-3,29e27f62701248f1,de7fdf8593b96f4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:21.355 INFO 26087 --- [prometheus-minute-schedule-1,dee5029b67ecd887,3ca83c10b9fda83c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:22.594 WARN 26087 --- [ocp-async-5,c50a8b7c590981fb,d31d35a1fe4aa607] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:22.594 INFO 26087 --- [ocp-async-3,23156df9daa76f92,a0712e787b690cd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:23.276 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:23.277 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:24.355 INFO 26087 --- [prometheus-minute-schedule-1,48939358029a5636,d9bb0219dc0fce89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:24.594 INFO 26087 --- [ocp-async-3,2a98b8f8fac8626e,a9f54adbe66e9c44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:24.594 WARN 26087 --- [ocp-async-2,c4cd03a8a6ed30f6,77e2b08cbc323d82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:26.277 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:26.277 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:26.525 INFO 26087 --- [ocp-schedules-14,f20ae956e47825f0,5ed7f03852bb441a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:14:26.525 INFO 26087 --- [ocp-schedules-9,557e2668ba73f5e1,f5d1eabebe25a6ab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:14:26.548 INFO 26087 --- [ocp-async-1,23fa94a645f2a4d0,83236de50c3f52a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:14:26.563 INFO 26087 --- [ocp-schedules-10,981b4f0e6959382a,766ec6da52689f4b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:14:26.564 INFO 26087 --- [ocp-schedules-10,981b4f0e6959382a,766ec6da52689f4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:26.564 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:14:26.595 INFO 26087 --- [ocp-async-7,e213f3722e93f4fc,7c480bdd754c8707] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:26.595 WARN 26087 --- [ocp-async-8,80321af4042148af,14d5dd549ec29d74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:14:26.622848+08:00, lastProcessTime=2025-04-08T21:14:16.623089+08:00 2025-04-08 21:14:26.634 INFO 26087 --- [ocp-schedules-24,7597ad44d92ba6b2,cec27a0a974f39c9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:14:26.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:14:26.622848+08:00 2025-04-08 21:14:27.355 INFO 26087 --- [prometheus-minute-schedule-1,308d43ac76a6f46a,06d3ee89b86f4c69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:27.526 INFO 26087 --- [ocp-schedules-28,59974b9c710b1531,2c37f7ea8251217f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:14:27.527 INFO 26087 --- [ocp-schedules-28,59974b9c710b1531,2c37f7ea8251217f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:14:28.595 WARN 26087 --- [ocp-async-2,9480c03e1054c57a,47b1dfd75bf8c698] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:28.595 INFO 26087 --- [ocp-async-9,923104b73ac0e7b5,67f1bb6e335281c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:28.991 INFO 26087 --- [ocp-schedules-17,433634a84d5f3d40,28efd8ebf7854e07] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:14:29.277 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:29.278 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:30.355 INFO 26087 --- [prometheus-minute-schedule-1,d696e3e0f2184431,973e9b1449edf87e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:30.595 WARN 26087 --- [ocp-async-3,01c3e8768334445b,3f371a9fb0e24017] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:30.595 INFO 26087 --- [ocp-async-10,17a2601b35014322,b5442a97aac48a3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:32.278 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:32.279 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:32.595 WARN 26087 --- [ocp-async-10,662b2680790773b8,e1a741cf32bac4bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:32.595 INFO 26087 --- [ocp-async-5,2642115d9b0c4038,64d43895920bd9fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:33.355 INFO 26087 --- [prometheus-minute-schedule-1,de3f2358b3bf9198,c11de0e8e9cbebce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:34.595 WARN 26087 --- [ocp-async-7,d01480aac5c391e1,728228cfb9ebde42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:34.595 INFO 26087 --- [ocp-async-2,756662be200a5912,1fa9b9a19398017f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:35.279 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:35.280 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:36.355 INFO 26087 --- [prometheus-minute-schedule-1,4a8678d090c2e00b,60ccf748ec862494] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:36.548 INFO 26087 --- [ocp-async-2,ca46ad63bb4f4533,d22c21a04f64271d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:14:36.565 INFO 26087 --- [ocp-schedules-10,981b4f0e6959382a,766ec6da52689f4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:14:36.595 WARN 26087 --- [ocp-async-6,6c32f60a1fd5f304,5165e1d5ad54c058] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:36.595 INFO 26087 --- [ocp-async-1,9fb820f9ad255db4,ec15654e452136f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:14:36.623424+08:00, lastProcessTime=2025-04-08T21:14:26.622848+08:00 2025-04-08 21:14:36.635 INFO 26087 --- [ocp-schedules-4,b0987bee2c3ff982,6873edc69ecb4042] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:14:36.745 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:14:36.623424+08:00 2025-04-08 21:14:37.109 INFO 26087 --- [ocp-schedules-19,461902a7f0b280a0,48271a713ee95280] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:14:37.114 INFO 26087 --- [ocp-schedules-19,461902a7f0b280a0,48271a713ee95280] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:14:37.114 INFO 26087 --- [ocp-schedules-19,461902a7f0b280a0,48271a713ee95280] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:14:37.540 INFO 26087 --- [ocp-schedules-8,93231d3623c1b620,fa221eb8210bb3b3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:14:37.541 INFO 26087 --- [ocp-schedules-8,93231d3623c1b620,fa221eb8210bb3b3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:14:38.280 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:38.280 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:38.595 WARN 26087 --- [ocp-async-10,177e1ec91569bb6c,b6f0c22f8204135b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:38.595 INFO 26087 --- [ocp-async-8,113dec722a0680da,92e3c509a64a2410] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:39.012 INFO 26087 --- [ocp-schedules-29,4a87ccac9c3e1801,381757c11f67c8a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:14:39.355 INFO 26087 --- [prometheus-minute-schedule-1,4a07b39a51380a47,cddcb8b0f0a1591d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:40.001 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:14:40.001 INFO 26087 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:14:40.596 WARN 26087 --- [ocp-async-4,9fd0be79c1502a09,ea045ae5ad042453] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:40.596 INFO 26087 --- [ocp-async-9,7f9fbb0e6f74f5f0,0461652781fac481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:41.280 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:41.281 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:42.355 INFO 26087 --- [prometheus-minute-schedule-1,fe0d4e886c66d2c6,6b9d70e813c322ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:42.596 WARN 26087 --- [ocp-async-2,be6f4effab1869fc,c33195e504b2bacb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:42.596 INFO 26087 --- [ocp-async-6,b7e1deb313bf8a2c,f9f8dc1822217e56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:44.281 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:44.282 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:44.596 INFO 26087 --- [ocp-async-6,6e216f50caae8f09,162dda033535960c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:44.596 WARN 26087 --- [ocp-async-5,0eaa3c36c48e2e05,22f81647cabae30f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:45.355 INFO 26087 --- [prometheus-minute-schedule-1,a92c45f4548bf4fc,925346add9c516e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:46.521 INFO 26087 --- [ocp-schedules-18,9c0f33fd3ecb04e7,5d731a16c748bff2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:14:46.521 INFO 26087 --- [ocp-schedules-28,17b3e8a897193b60,3324e771cc47eaed] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:14:46.522 INFO 26087 --- [ocp-schedules-28,17b3e8a897193b60,3324e771cc47eaed] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:14:46.523 INFO 26087 --- [ocp-schedules-6,1cc2b301bb65a613,200e3c37faf779ae] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:14:46.523 INFO 26087 --- [ocp-schedules-6,1cc2b301bb65a613,200e3c37faf779ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:14:46.523 INFO 26087 --- [ocp-schedules-13,cf3adc74dd4a4a75,1d372e02c0b5d91c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:14:46.523 INFO 26087 --- [ocp-schedules-13,cf3adc74dd4a4a75,1d372e02c0b5d91c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:14:46.524 INFO 26087 --- [ocp-schedules-25,1a5b2b14503ea650,11afc0d7356f829c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:14:46.524 INFO 26087 --- [ocp-schedules-3,be203add8c9d98fd,2f85c02ccfd0c090] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:14:46.525 INFO 26087 --- [ocp-schedules-25,1a5b2b14503ea650,11afc0d7356f829c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:46.525 INFO 26087 --- [ocp-schedules-3,be203add8c9d98fd,2f85c02ccfd0c090] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:14:46.526 INFO 26087 --- [ocp-schedules-4,cacc1d3db2a2a500,63935499a2fb5b90] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:14:46.548 INFO 26087 --- [ocp-async-6,cc5bd1cb042722b0,eb0238275d592a9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:14:46.596 WARN 26087 --- [ocp-async-7,964d39f8596cd1b3,0d504423d17b8665] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:46.596 INFO 26087 --- [ocp-async-4,5cb44fdf9f5dae41,b470a3753eab2cbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:46.597 INFO 26087 --- [ocp-schedules-16,df6b28cd80054c93,3e50561f08e0923e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:14:46.597 INFO 26087 --- [ocp-schedules-16,7bfa96aed41c6611,eb2addb18c32a2b2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:14:46.597 INFO 26087 --- [ocp-schedules-16,7bfa96aed41c6611,eb2addb18c32a2b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:14:46.598 INFO 26087 --- [ocp-schedules-29,894c5b3ae3346bee,73b6d472d5f25b3f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:14:46.604 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:14:46.604 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:14:46.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:14:46.624255+08:00, lastProcessTime=2025-04-08T21:14:36.623424+08:00 2025-04-08 21:14:46.636 INFO 26087 --- [ocp-schedules-21,b7adfa0436b6f73a,32b69212804e0c74] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:14:46.720 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:14:46.624255+08:00 2025-04-08 21:14:47.282 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:47.283 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:47.609 INFO 26087 --- [ocp-schedules-5,6c8693b4f7c5ced0,5054f712f077e5a1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:14:47.611 INFO 26087 --- [ocp-schedules-5,6c8693b4f7c5ced0,5054f712f077e5a1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:14:48.355 INFO 26087 --- [prometheus-minute-schedule-1,1118fe5fbc193f0a,069eef8828e6c226] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:48.596 WARN 26087 --- [ocp-async-1,4c77e1caf36a73e0,d4fdcf649318ccb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:48.596 INFO 26087 --- [ocp-async-2,dac7a045f7b6d74b,ece709543e9aa0d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:49.040 INFO 26087 --- [ocp-schedules-28,babe1ab254acae1f,67d634115324c937] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:14:50.283 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:50.283 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:50.596 WARN 26087 --- [ocp-async-8,606550324949f6fa,aaaeb5446d8dabff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:50.596 INFO 26087 --- [ocp-async-5,4aece63e57bc826d,a25fc74a14667c37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:51.355 INFO 26087 --- [prometheus-minute-schedule-1,c29277dbf43d7a97,ac70c6fd5a6cb323] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:52.596 WARN 26087 --- [ocp-async-9,7a2837b130c888ad,801fceac073b87fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:52.597 INFO 26087 --- [ocp-async-6,2ce47382258ceefb,15adc301fbda998d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:53.283 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:53.284 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:54.355 INFO 26087 --- [prometheus-minute-schedule-1,5ac8e4b8ef165e3b,6155fc6be3dc3970] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:54.597 WARN 26087 --- [ocp-async-3,1cc5d7e458b74563,7790340a85e56af1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:54.597 INFO 26087 --- [ocp-async-1,27af1e7c8bfc3f5c,430486ea0bbabbde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:56.284 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:56.285 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:14:56.526 INFO 26087 --- [ocp-schedules-3,be203add8c9d98fd,2f85c02ccfd0c090] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:14:56.526 INFO 26087 --- [ocp-schedules-25,1a5b2b14503ea650,11afc0d7356f829c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:14:56.548 INFO 26087 --- [ocp-async-1,e81fc4965fb42f08,aa7ef38648966d45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:14:56.597 INFO 26087 --- [ocp-async-8,e643af9f2e1895d0,0c0246984123cd86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:56.597 WARN 26087 --- [ocp-async-5,d109dcdfdc151167,c38f3b0ee33476f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:14:56.624332+08:00, lastProcessTime=2025-04-08T21:14:46.624255+08:00 2025-04-08 21:14:56.637 INFO 26087 --- [ocp-schedules-2,a2b73f3b47c9c318,305bb22873b1bf1f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:14:56.756 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:14:56.624332+08:00 2025-04-08 21:14:57.115 INFO 26087 --- [ocp-schedules-19,8d7311a4d474f566,8773d5f73fd3d703] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:14:57.136 INFO 26087 --- [ocp-schedules-19,8d7311a4d474f566,8773d5f73fd3d703] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:14:57.136 INFO 26087 --- [ocp-schedules-19,8d7311a4d474f566,8773d5f73fd3d703] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:14:57.355 INFO 26087 --- [prometheus-minute-schedule-1,1ceb29bb15016f22,0060f38a15ba441d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118100, interval=60 2025-04-08 21:14:57.684 INFO 26087 --- [ocp-schedules-28,614585301f8ced6d,0205d397424fa935] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:14:57.686 INFO 26087 --- [ocp-schedules-28,614585301f8ced6d,0205d397424fa935] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:14:58.597 WARN 26087 --- [ocp-async-7,71c29f5cf9991ca2,b046d540aa5de065] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:14:58.597 INFO 26087 --- [ocp-async-6,f856755e6a08708d,7e8e3a3db50d411b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:14:59.061 INFO 26087 --- [ocp-schedules-30,1fd97c6c3e5132ff,2834b7db99df01b9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:14:59.285 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:14:59.285 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:00.355 INFO 26087 --- [prometheus-minute-schedule-1,4cbd08a18f2806b3,230ac72531d298cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:00.388 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:10:01.0, nextRun=2025-04-08 21:15:00.0, currentDate=Tue Apr 08 21:15:00 CST 2025 2025-04-08 21:15:00.388 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:10:01.0, nextRunTime=2025-04-08 21:15:00.0} 2025-04-08 21:15:00.390 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:15:00.390 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:15:00.390+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:15:00.408 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:10:01.0, nextRun=2025-04-08 21:15:00.0, currentDate=Tue Apr 08 21:15:00 CST 2025 2025-04-08 21:15:00.408 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:10:01.0, nextRunTime=2025-04-08 21:15:00.0} 2025-04-08 21:15:00.414 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:15:00.416 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:15:00.417 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,a0d2dc42d29b7e0f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:15:00.597 INFO 26087 --- [ocp-async-3,2b2a5ca083f0ba40,861e5703a02cb645] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:00.597 WARN 26087 --- [ocp-async-4,dfd23c0191a415a9,eac183970c0b4256] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:01.367 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,1374a0b9150b3c21] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000119, operation:EXECUTE, state: RUNNING 2025-04-08 21:15:01.399 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:15:01.400 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000119, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000010, task_operation=execute, latest_execution_start_time=2025-04-08T21:15:01.363+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000119}, listMap={}}, executor=192.168.1.61 2025-04-08 21:15:01.408 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.409 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for sgkydb 2025-04-08 21:15:01.410 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.410 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.410 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.411 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.411 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.412 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.412 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:sgkydb 2025-04-08 21:15:01.418 INFO 26087 --- [subtask-executor7,8ec1a44aeefd3a62,6c614798f1b2443c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000119, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:15:01.811 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000010, status:SUCCESS 2025-04-08 21:15:01.811 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000010, status:SUCCESS 2025-04-08 21:15:02.286 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:02.286 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:02.597 WARN 26087 --- [ocp-async-5,4dc9b706a2912146,69dee679e3cda770] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:02.597 INFO 26087 --- [ocp-async-10,5a653f6cb02e726f,543df9e533e87cff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:03.355 INFO 26087 --- [prometheus-minute-schedule-1,6e01f465e1161236,c1395723a34258a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:04.597 WARN 26087 --- [ocp-async-7,288f8d5f72b4f04d,85afa32293fed987] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:04.597 INFO 26087 --- [ocp-async-9,eaca0aa2cd46f6cc,65856e8bca957c83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:05.286 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:05.287 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:06.355 INFO 26087 --- [prometheus-minute-schedule-1,c353f1d747454bcc,609b102c8e2ed205] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:06.355 INFO 26087 --- [prometheus-cache-monitor1,35fa704164075688,709c74ba2c2faad6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1496,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:15:06.355 INFO 26087 --- [prometheus-cache-monitor1,35fa704164075688,709c74ba2c2faad6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:15:06.549 INFO 26087 --- [ocp-async-9,e3694964c0a64ea3,4b86fcb13348fcbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:15:06.598 WARN 26087 --- [ocp-async-3,6f8f5286d941f4d3,1439d5a7c84c998c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:06.598 INFO 26087 --- [ocp-async-2,7dc3e4309bec9dff,ab3e5bbe071cbf38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:15:06.623487+08:00, lastProcessTime=2025-04-08T21:14:56.624332+08:00 2025-04-08 21:15:06.638 INFO 26087 --- [ocp-schedules-27,833445c4c9d4c629,65ec8f5330dedf4f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:15:06.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:15:06.623487+08:00 2025-04-08 21:15:07.702 INFO 26087 --- [ocp-schedules-22,491be5f72bcb5684,fd8a76ba7dff644c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:15:07.704 INFO 26087 --- [ocp-schedules-22,491be5f72bcb5684,fd8a76ba7dff644c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:15:08.287 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:08.288 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:08.598 WARN 26087 --- [ocp-async-1,5e061266bcb3d532,6d8f928b57727d5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:08.598 INFO 26087 --- [ocp-async-5,785abf6570c6cd14,4e566463f17d2a59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:09.089 INFO 26087 --- [ocp-schedules-7,f2c3f849d20b25ee,5b49a5616a04d6dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:15:09.355 INFO 26087 --- [prometheus-minute-schedule-1,04624ccbfc9d12a9,43c5d24f9777fc8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:10.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:15:10.002 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:15:10.598 WARN 26087 --- [ocp-async-8,cdecf5b21667e674,6e41855ef4559efb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:10.598 INFO 26087 --- [ocp-async-7,cd72328b1eb62840,6f404e26ce0f3d8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:15:11.288 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:11.288 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:12.355 INFO 26087 --- [prometheus-minute-schedule-1,923098c6942200dd,e8a7fcb711f98648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:12.405 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118040, nodeCount=374, affectRows=374 2025-04-08 21:15:12.407 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118040, nodeCount=371, validCount=371 2025-04-08 21:15:12.598 WARN 26087 --- [ocp-async-6,0801845ba1379092,95074e15e625a748] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:12.598 INFO 26087 --- [ocp-async-4,fccca077809e81b9,b5abbe88074808cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:14.288 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:14.289 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:14.598 WARN 26087 --- [ocp-async-1,01d24d567ff6a665,de0a8d9333746634] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:14.598 INFO 26087 --- [ocp-async-4,b43019cef950c9be,b3731aceeb83165f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:15.355 INFO 26087 --- [prometheus-minute-schedule-1,a461b90a1b9a6059,b545c441794b2383] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:16.514 INFO 26087 --- [ocp-schedules-30,197105df8c589925,3ba8c99bac461053] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:15:16.514+08:00, tenantSize=0 2025-04-08 21:15:16.517 INFO 26087 --- [ocp-schedules-14,0016c524f2e20165,acd7792a07e7a51e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:15:16.517 INFO 26087 --- [ocp-schedules-2,795ecb2500911991,7578a53dd3fa0706] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:15:16.520 INFO 26087 --- [ocp-schedules-14,0016c524f2e20165,acd7792a07e7a51e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:15:16.521 INFO 26087 --- [ocp-schedules-18,1832f54257f41a69,0d7fff110d812087] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:15:16.521 INFO 26087 --- [ocp-schedules-18,1832f54257f41a69,0d7fff110d812087] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:15:16.521 INFO 26087 --- [ocp-schedules-27,4bc477f02489366a,886323603c881313] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:15:16.523 INFO 26087 --- [ocp-schedules-26,341098f885b76cba,9304038a21938f3d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:15:16.523 INFO 26087 --- [ocp-schedules-26,341098f885b76cba,9304038a21938f3d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:15:16.523 INFO 26087 --- [ocp-schedules-28,209e9381c95a25dd,f1941a9db3dd35a7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:15:16.523 INFO 26087 --- [ocp-schedules-28,209e9381c95a25dd,f1941a9db3dd35a7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:15:16.524 INFO 26087 --- [ocp-schedules-2,795ecb2500911991,7578a53dd3fa0706] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:15:16.524 INFO 26087 --- [ocp-schedules-1,ac1af4ba4aa4c53b,a20c5cf352135e8a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:15:16.524 INFO 26087 --- [ocp-schedules-8,15f1c841e0ef4633,c772624c6c5b0688] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:15:16.525 INFO 26087 --- [ocp-schedules-1,ac1af4ba4aa4c53b,a20c5cf352135e8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:16.525 INFO 26087 --- [ocp-schedules-8,15f1c841e0ef4633,c772624c6c5b0688] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:15:16.526 INFO 26087 --- [ocp-schedules-18,b680a4b2250285c4,9e457d96a6fcb9ae] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:15:16.549 INFO 26087 --- [ocp-async-8,9b9786a29c2cca80,35bc8d0863abc7bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:15:16.558 INFO 26087 --- [ocp-schedules-17,889f55fdbe6a8400,aac5b74146ca8868] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 21:15:16.558 INFO 26087 --- [ocp-schedules-17,889f55fdbe6a8400,aac5b74146ca8868] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 21:15:16.585 INFO 26087 --- [ocp-schedules-22,3ec3ddea8554df0e,21abfb345bf1e6b4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1496,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":372,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:15:16.585 INFO 26087 --- [ocp-schedules-22,3ec3ddea8554df0e,21abfb345bf1e6b4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:15:16.586 INFO 26087 --- [ocp-schedules-7,13d901768ab72c91,32e4d4fb57c15716] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:15:16.598 WARN 26087 --- [ocp-async-9,7f5d89f9dec287d3,6b8d0c02cf863087] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:16.598 INFO 26087 --- [ocp-async-6,93fdb4b42c8badbc,a1fa068a2cf5f27e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:16.599 INFO 26087 --- [ocp-schedules-30,8bd490794ef4c565,c26283d853d5d933] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:15:16.604 INFO 26087 --- [ocp-schedules-16,7bfa96aed41c6611,eb2addb18c32a2b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:15:16.604 INFO 26087 --- [ocp-schedules-16,1a84c550ccd38625,e94ae85067581d4f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:15:16.605 INFO 26087 --- [ocp-schedules-16,1a84c550ccd38625,e94ae85067581d4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:16.611 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:15:16.611 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:15:16.613 INFO 26087 --- [ocp-schedules-28,83ab0a18f8e60bac,be3b9f2b516c079c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:15:16.613 INFO 26087 --- [ocp-schedules-28,83ab0a18f8e60bac,be3b9f2b516c079c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:15:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:15:16.623392+08:00, lastProcessTime=2025-04-08T21:15:06.623487+08:00 2025-04-08 21:15:16.631 INFO 26087 --- [ocp-schedules-14,4b1baf0887fb868d,3e712a721e06724c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:15:16.631 INFO 26087 --- [ocp-schedules-21,a69d2745d997d963,d14af98883994524] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:15:16.631+08:00, clusters=[1000002] 2025-04-08 21:15:16.631 INFO 26087 --- [ocp-schedules-21,a69d2745d997d963,d14af98883994524] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:15:16.635 INFO 26087 --- [ocp-schedules-6,8a4123cb77751620,6c4cf1e3a5c5c816] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:15:16.635 INFO 26087 --- [ocp-schedules-6,8a4123cb77751620,6c4cf1e3a5c5c816] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:15:16.635 INFO 26087 --- [ocp-schedules-11,4b4fd0531ae8eec3,a99a2eec0c6dfa95] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:15:16.635+08:00, clusters=[1000002] 2025-04-08 21:15:16.637 INFO 26087 --- [ocp-schedules-11,4b4fd0531ae8eec3,a99a2eec0c6dfa95] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:15:16.639 INFO 26087 --- [ocp-schedules-9,eb320560eb93d29b,9d172484c7eb4562] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:15:16.651 INFO 26087 --- [ocp-schedules-19,0528ab5314144257,6e8c97e10cd09958] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:15:16.651 INFO 26087 --- [ocp-schedules-19,0528ab5314144257,6e8c97e10cd09958] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:15:16.657 INFO 26087 --- [ocp-schedules-19,0528ab5314144257,6e8c97e10cd09958] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:15:16.657 INFO 26087 --- [ocp-schedules-19,0528ab5314144257,6e8c97e10cd09958] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:15:16.665 INFO 26087 --- [ocp-schedules-13,d20c2184b333658c,c3e3ce2101799e64] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:15:16.667 INFO 26087 --- [ocp-schedules-13,d20c2184b333658c,c3e3ce2101799e64] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:15:16.667 INFO 26087 --- [ocp-schedules-13,d20c2184b333658c,c3e3ce2101799e64] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:15:16.717 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:15:16.623392+08:00 2025-04-08 21:15:16.723 INFO 26087 --- [ocp-schedules-20,0cfe319eb176d210,546b918f25cd5197] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:15:16.793 INFO 26087 --- [ocp-schedules-3,8315457b6f720f5c,5b8a2b3f361bbf4e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:15:16.793 INFO 26087 --- [ocp-schedules-3,8315457b6f720f5c,5b8a2b3f361bbf4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:16.797 INFO 26087 --- [ocp-schedules-23,b6e3ed721b12e1f3,07033e0285ba9d6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:15:16.812 INFO 26087 --- [ocp-schedules-25,33f91b0d5f7dbbec,31c8cb893e0dc3f8] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:15:16.813 INFO 26087 --- [ocp-schedules-25,33f91b0d5f7dbbec,31c8cb893e0dc3f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:16.813 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:15:16.814 ERROR 26087 --- [ocp-schedules-3,8315457b6f720f5c,5b8a2b3f361bbf4e] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:15:16.814 INFO 26087 --- [ocp-schedules-3,8315457b6f720f5c,5b8a2b3f361bbf4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:15:16.826 INFO 26087 --- [ocp-schedules-5,15711bff9e35ef45,e160404ae7ba3870] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:15:16.826 INFO 26087 --- [ocp-schedules-5,15711bff9e35ef45,e160404ae7ba3870] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:15:16.826 INFO 26087 --- [ocp-schedules-5,15711bff9e35ef45,e160404ae7ba3870] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:15:16.826 INFO 26087 --- [ocp-schedules-5,15711bff9e35ef45,e160404ae7ba3870] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208338840276992 2025-04-08 21:15:16.826 INFO 26087 --- [ocp-schedules-5,15711bff9e35ef45,e160404ae7ba3870] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:15:16.826 INFO 26087 --- [ocp-schedules-5,15711bff9e35ef45,e160404ae7ba3870] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:15:16.829 ERROR 26087 --- [ocp-schedules-25,33f91b0d5f7dbbec,31c8cb893e0dc3f8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=sgkydb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:15:16.829 INFO 26087 --- [ocp-schedules-25,33f91b0d5f7dbbec,31c8cb893e0dc3f8] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:15:16.879 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:15:16.879 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:15:16.880 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.880 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:15:16.881 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.881 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.881 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.881 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:15:16.893 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.893 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.893 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.894 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:15:16.905 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.905 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.905 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.905 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:15:16.916 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.916 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.916 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.917 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:15:16.928 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.928 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.929 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.929 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:15:16.933 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.934 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.934 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.934 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:15:16.937 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.938 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.938 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.938 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:15:16.942 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.942 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.942 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.942 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:15:16.947 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.947 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.947 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.947 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:15:16.951 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.952 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.952 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.952 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:15:16.956 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.956 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.956 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.957 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:15:16.963 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.963 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.964 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.964 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:15:16.974 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.974 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.975 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.975 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:15:16.979 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.979 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.980 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.980 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:15:16.984 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.984 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.985 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.985 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:15:16.989 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.989 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.990 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.990 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:15:16.993 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.993 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.994 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.994 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:15:16.997 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.998 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:16.998 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:16.998 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:15:17.001 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.002 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.002 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.002 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:15:17.006 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.007 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.007 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.007 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:15:17.011 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.011 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.011 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.012 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:15:17.015 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.015 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.016 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.016 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:15:17.019 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.019 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.020 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.020 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:15:17.023 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.023 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.024 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.024 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:15:17.027 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.027 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.027 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.027 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:15:17.031 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.031 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.031 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.031 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:15:17.035 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.035 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.035 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.035 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:15:17.039 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.039 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.039 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.039 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:15:17.042 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.042 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.042 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.043 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:15:17.046 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.046 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.046 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.046 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:15:17.049 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.049 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.050 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.050 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:15:17.053 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.053 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.054 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.054 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:15:17.057 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.057 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.058 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.058 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:15:17.061 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.061 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.062 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.062 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:15:17.067 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.067 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.068 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.068 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:15:17.072 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.072 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.072 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.073 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:15:17.076 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.076 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.077 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.077 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:15:17.081 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.081 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.081 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.081 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:15:17.084 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.084 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.084 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.084 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:15:17.087 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.087 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.087 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.087 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:15:17.090 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.090 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.090 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.091 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:15:17.093 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.093 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.094 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.094 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:15:17.102 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.102 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.102 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.102 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:15:17.119 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.120 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.120 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.120 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:15:17.125 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.126 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.126 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.126 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:15:17.131 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.131 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.131 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.132 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:15:17.136 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.136 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.136 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.137 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:15:17.137 INFO 26087 --- [ocp-schedules-10,6136326f199becbd,0a090480d29ecda2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:15:17.140 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.140 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.140 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.140 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:15:17.143 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.143 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.144 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.144 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:15:17.147 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:15:17.147 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:15:17.147 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.147 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:15:17.148 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.148 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.149 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.149 INFO 26087 --- [ocp-schedules-10,6136326f199becbd,0a090480d29ecda2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:15:17.149 INFO 26087 --- [ocp-schedules-10,6136326f199becbd,0a090480d29ecda2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:15:17.149 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:15:17.153 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.153 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:15:17.153 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:17.153 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:15:17.156 INFO 26087 --- [ocp-schedules-22,031750c3efdfd091,e1c3869ecdf2cd1b] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:15:17.289 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:17.290 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:17.511 INFO 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:15:17.511 INFO 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:15:17.511 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:15:17.511 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:15:17.511 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:15:17.514 INFO 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:15:17.514 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:15:17.514 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:15:17.515 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:15:17.515 WARN 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:15:17.515 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:15:17.515 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:15:17.515 INFO 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:15:17.515 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:15:17.515 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:15:17.516 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:15:17.516 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:15:17.516 WARN 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:15:17.517 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:15:17.517 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:15:17.517 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:15:17.520 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:15:17.520 INFO 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:15:17.520 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:15:17.521 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:15:17.521 WARN 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:15:17.524 ERROR 26087 --- [ocp-schedules-6,dff2c0dcf14165ab,15877993c16ab66c] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2025-04-08 21:15:17.706 INFO 26087 --- [ocp-schedules-17,7dc678d3468d83bc,03bed1b77fd49e39] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:15:17.707 INFO 26087 --- [ocp-schedules-17,7dc678d3468d83bc,03bed1b77fd49e39] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:15:18.355 INFO 26087 --- [prometheus-minute-schedule-1,5ec048b5c4b70ba4,da059c6f560da2cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:18.598 WARN 26087 --- [ocp-async-6,4d36e7dd869c1951,1a528191ce1d170d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:18.598 INFO 26087 --- [ocp-async-1,02ffe735ca968746,6adfcf85c5a6d1a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:19.110 INFO 26087 --- [ocp-schedules-19,64e3c67619397691,be66131a8e74f2bf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:15:20.290 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:20.290 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:15:20.598 WARN 26087 --- [ocp-async-1,19d1788089205c6c,74682cc4e8e2b751] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:20.599 INFO 26087 --- [ocp-async-4,0aa4483b76b7e89e,facfa3781bda8bae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:21.355 INFO 26087 --- [prometheus-minute-schedule-1,9c89a9bfa25276b6,0bdf32ba32e02636] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:21.707 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:15:21.707 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:15:21.710 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:15:21.710 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:15:21.710 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:15:21.710 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:15:21.714 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:15:21.714 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:15:21.714 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:15:21.714 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:15:21.738 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:15:21.744 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:15:21.744 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:15:21.744 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:15:21.744 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:15:21.744 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:15:21.749 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:15:22.054 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:15:22.383 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.668633509S 2025-04-08 21:15:22.383 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:15:22.439 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:15:22.439 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:15:22.464 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:15:22.508 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:15:22.508 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:15:22.508 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:15:22.508 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:15:22.508 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:15:22.508 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:15:22.514 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:15:22.599 WARN 26087 --- [ocp-async-3,9c33a0979f344daa,f560d39d9f7c4dc1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:22.599 INFO 26087 --- [ocp-async-4,103d20633e9535af,79769828a52daac6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:22.610 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] o.s.boot.SpringApplication : Started application in 0.899 seconds (JVM running for 827.983) 2025-04-08 21:15:22.610 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.899579339S 2025-04-08 21:15:22.885 INFO 26087 --- [ocp-schedules-24,3090372217f899de,a781540cb8309171] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:15:23.291 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:23.291 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:24.355 INFO 26087 --- [prometheus-minute-schedule-1,a556329abcefdde6,c3039157fc32d98e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:24.599 WARN 26087 --- [ocp-async-2,d4613a8a5eb1002c,68297c8da55bcbe4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:24.599 INFO 26087 --- [ocp-async-6,586e217e72192ced,092196769b5869b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:26.291 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:26.292 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:26.526 INFO 26087 --- [ocp-schedules-8,15f1c841e0ef4633,c772624c6c5b0688] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:15:26.526 INFO 26087 --- [ocp-schedules-1,ac1af4ba4aa4c53b,a20c5cf352135e8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:15:26.549 INFO 26087 --- [ocp-async-1,e427145e44d6b662,6c86ac52406c8401] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:15:26.599 WARN 26087 --- [ocp-async-3,385212d9f7b29026,23e729d9535bbeb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:26.599 INFO 26087 --- [ocp-async-5,ae7a2019d1fba65b,72518d888c1b3864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:15:26.623515+08:00, lastProcessTime=2025-04-08T21:15:16.623392+08:00 2025-04-08 21:15:26.641 INFO 26087 --- [ocp-schedules-3,82861b964e654b94,0baf6af4da8e80bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:15:26.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:15:26.623515+08:00 2025-04-08 21:15:27.355 INFO 26087 --- [prometheus-minute-schedule-1,dbe240a77be4d3de,26fb41e51bf64c27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:27.728 INFO 26087 --- [ocp-schedules-4,357ab19c8c29e452,96d543818663542a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:15:27.729 INFO 26087 --- [ocp-schedules-4,357ab19c8c29e452,96d543818663542a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:15:28.599 WARN 26087 --- [ocp-async-7,54f50ef6d0a92e06,153c9002a7c0586d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:28.599 INFO 26087 --- [ocp-async-2,45393002d90fc835,d48f034cf2538704] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:29.138 INFO 26087 --- [ocp-schedules-24,9523e317eca36f65,acbb5708715e583d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:15:29.292 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:29.293 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:30.355 INFO 26087 --- [prometheus-minute-schedule-1,4e91f95d5bdbdb5d,a64436cc8663eca8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:30.599 WARN 26087 --- [ocp-async-10,283fcb16497f0f24,b88a1faf53aaaa18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:30.599 INFO 26087 --- [ocp-async-4,7272d65363d92b5d,2d7c6dd0904d5560] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:32.293 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:32.293 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:32.599 WARN 26087 --- [ocp-async-1,e652424eab50f043,3bcd22d2ef54b57a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:32.599 INFO 26087 --- [ocp-async-9,572304b73e75f90b,3d607258e891851b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:33.355 INFO 26087 --- [prometheus-minute-schedule-1,f3e774a4465f8c22,9f232b24d5d6b3cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:34.599 WARN 26087 --- [ocp-async-5,4a10613c4f8811e8,a23aa9374792b8db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:34.599 INFO 26087 --- [ocp-async-7,65661c08fe750098,1e2050e52e07b21d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:34.995 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:15:34.996 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:15:34.996 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:15:34.996 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:15:35.002 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster sgkydb(1000002), status=TAKINGOVER 2025-04-08 21:15:35.002 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=6ms, period=120000ms 2025-04-08 21:15:35.294 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:35.294 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:36.355 INFO 26087 --- [prometheus-minute-schedule-1,1439c6f51d37cdb4,3deed28c57564331] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:36.549 INFO 26087 --- [ocp-async-7,13934e005fd3cf32,b0122a5e0ac6984b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:15:36.565 INFO 26087 --- [ocp-schedules-22,095f744602b922a3,95293a8aca1c011d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:15:36.565 INFO 26087 --- [ocp-schedules-22,095f744602b922a3,95293a8aca1c011d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:36.566 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:15:36.600 WARN 26087 --- [ocp-async-6,717aa78039c97303,bd388837ebdbfce0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:36.600 INFO 26087 --- [ocp-async-10,c4d80dde9884a528,dbd00fca5f91f205] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:15:36.623606+08:00, lastProcessTime=2025-04-08T21:15:26.623515+08:00 2025-04-08 21:15:36.642 INFO 26087 --- [ocp-schedules-25,90f2cf1a87bc9911,be946b42a73cba61] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:15:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:15:36.623606+08:00 2025-04-08 21:15:37.150 INFO 26087 --- [ocp-schedules-24,2f99e8dcd8bfca75,6c5293bb155fa8de] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:15:37.154 INFO 26087 --- [ocp-schedules-24,2f99e8dcd8bfca75,6c5293bb155fa8de] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:15:37.155 INFO 26087 --- [ocp-schedules-24,2f99e8dcd8bfca75,6c5293bb155fa8de] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:15:37.772 INFO 26087 --- [ocp-schedules-29,367f2733b93dcd05,91214d6d249e0466] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:15:37.773 INFO 26087 --- [ocp-schedules-29,367f2733b93dcd05,91214d6d249e0466] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:15:38.294 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:38.295 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:38.600 WARN 26087 --- [ocp-async-1,6c44dd4069bdd793,5201544f8b168b8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:38.600 INFO 26087 --- [ocp-async-9,81836ec48b81ac78,adfa0f6d04b2ccef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:39.159 INFO 26087 --- [ocp-schedules-29,a3bbd73594363a39,2107bbe972605e78] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:15:39.355 INFO 26087 --- [prometheus-minute-schedule-1,50a86a0dc3d78268,9a642348f8db48d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:40.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:15:40.002 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:15:40.600 WARN 26087 --- [ocp-async-2,b40bfbd2a5bc5cdb,87ecd8846e846daf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:40.600 INFO 26087 --- [ocp-async-8,b5bd7a325db5512b,cff8f300fa512a11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:41.295 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:41.295 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:42.355 INFO 26087 --- [prometheus-minute-schedule-1,8c6e3285b265e4f7,9aebeb505a2c11d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:42.600 WARN 26087 --- [ocp-async-4,ac68145391cd2853,cea284b48ba4be2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:42.600 INFO 26087 --- [ocp-async-7,fc4362a1e936a7d2,14acd23ee9c7012c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:44.296 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:44.296 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:44.600 WARN 26087 --- [ocp-async-10,c6461fa505ba8ff5,6de75c4f07b9b938] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:44.600 INFO 26087 --- [ocp-async-3,5575f89e24e472b0,22797d1d76415ad8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:45.355 INFO 26087 --- [prometheus-minute-schedule-1,25ea61fbaafc738a,a94203bec70f51c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:46.521 INFO 26087 --- [ocp-schedules-9,3481b98a328332cc,4a8fdf5fc16327c8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:15:46.522 INFO 26087 --- [ocp-schedules-15,1a634924d50cc362,321f7f777b09d65e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:15:46.522 INFO 26087 --- [ocp-schedules-15,1a634924d50cc362,321f7f777b09d65e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:15:46.536 INFO 26087 --- [ocp-schedules-29,24fdf14b2086a614,6106008204b03b01] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:15:46.536 INFO 26087 --- [ocp-schedules-30,f382a6ada769ca13,fbc9f7790a862f69] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:15:46.537 INFO 26087 --- [ocp-schedules-30,f382a6ada769ca13,fbc9f7790a862f69] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:46.537 INFO 26087 --- [ocp-schedules-29,24fdf14b2086a614,6106008204b03b01] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:46.537 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:15:46.538 INFO 26087 --- [ocp-schedules-27,604a2646dad05a24,3680c957dc454059] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:15:46.538 INFO 26087 --- [ocp-schedules-27,604a2646dad05a24,3680c957dc454059] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:15:46.538 INFO 26087 --- [ocp-schedules-11,99f0599949a3fa68,43966e3c95231cb8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:15:46.538 INFO 26087 --- [ocp-schedules-11,99f0599949a3fa68,43966e3c95231cb8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:15:46.539 INFO 26087 --- [ocp-schedules-17,feb075aa1dab822d,21fe4d12a5c58367] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:15:46.549 INFO 26087 --- [ocp-async-3,aa7613e079ad8e65,938b840742bb9594] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:15:46.566 INFO 26087 --- [ocp-schedules-22,095f744602b922a3,95293a8aca1c011d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:15:46.600 WARN 26087 --- [ocp-async-8,6b0519572dd29ed8,8e86d2b07c245b60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:46.600 INFO 26087 --- [ocp-async-5,13828043b1cc31c9,396266397f70f02a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:46.601 INFO 26087 --- [ocp-schedules-24,46a064b6c005ab7d,cc3e275f7f7fb816] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:15:46.612 INFO 26087 --- [ocp-schedules-16,1a84c550ccd38625,e94ae85067581d4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:15:46.612 INFO 26087 --- [ocp-schedules-16,08eda80789c0e03c,7bb05dee7ce41161] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:15:46.613 INFO 26087 --- [ocp-schedules-16,08eda80789c0e03c,7bb05dee7ce41161] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:15:46.619 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:15:46.619 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:15:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:15:46.623382+08:00, lastProcessTime=2025-04-08T21:15:36.623606+08:00 2025-04-08 21:15:46.643 INFO 26087 --- [ocp-schedules-3,257ad3496d3304cd,7a4caf44fc450711] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:15:46.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:15:46.623382+08:00 2025-04-08 21:15:47.296 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:47.297 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:47.789 INFO 26087 --- [ocp-schedules-12,ab19ec11c7426f18,89c66eb9ce78c6a9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:15:47.791 INFO 26087 --- [ocp-schedules-12,ab19ec11c7426f18,89c66eb9ce78c6a9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:15:48.355 INFO 26087 --- [prometheus-minute-schedule-1,04c2bce0f391d287,cc727a5c7fd79cbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:48.600 WARN 26087 --- [ocp-async-7,f4c7e5fffae2fe2e,eec9c97085b24484] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:48.601 INFO 26087 --- [ocp-async-6,d4d385644be65c4b,e0424ce25e36225e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:49.189 INFO 26087 --- [ocp-schedules-6,117b54d2cffd6afb,4d1216ec25075405] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:15:50.297 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:50.298 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:50.601 WARN 26087 --- [ocp-async-1,9e2eddb2bd4dab15,a93a86d60320dd06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:50.601 INFO 26087 --- [ocp-async-9,8904fbb2ed739ce7,3097294262f62fa9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:51.355 INFO 26087 --- [prometheus-minute-schedule-1,80a5658298090c21,43ce9460697faa06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:51.939 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,1cc928d6f4217c76,4f3fc67669d6dcb3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments?taskInstanceId=1000007&subtaskInstanceId=1000061, client=10.11.12.102, traceId=1cc928d6f4217c76, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000007,1000061,] 2025-04-08 21:15:51.942 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,1cc928d6f4217c76,4f3fc67669d6dcb3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments client=10.11.12.102, traceId=1cc928d6f4217c76, duration=8 ms] 2025-04-08 21:15:52.601 WARN 26087 --- [ocp-async-2,18f6d61ccba59c2f,1cf5193af2791751] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:52.601 INFO 26087 --- [ocp-async-8,03081199fb582588,1ae5970b2f9e8660] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:53.298 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:53.299 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:53.624 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47f8ebd378eac8fc,0462de177acfbcab] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=47f8ebd378eac8fc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744118153621371,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 21:15:53.631 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47f8ebd378eac8fc,0462de177acfbcab] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000009, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:15:53.631 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47f8ebd378eac8fc,0462de177acfbcab] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=sgkydb, obClusterId=1744016068, rootServerJson={"ObCluster":"sgkydb","ObClusterId":1744016068,"ObRegion":"sgkydb","ObRegionId":1744016068,"ReadonlyRsList":[],"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1744118153625000}, groupName=sgkydb, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-04-08 21:15:53.631 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47f8ebd378eac8fc,0462de177acfbcab] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-04-08 21:15:53.631 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47f8ebd378eac8fc,0462de177acfbcab] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=sgkydb, clusterId=1744016068, affectRows=1 2025-04-08 21:15:53.637 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,47f8ebd378eac8fc,0462de177acfbcab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=192.168.1.62, traceId=47f8ebd378eac8fc, duration=15 ms] 2025-04-08 21:15:53.999 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [sgkydb] cache 2025-04-08 21:15:53.999 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:15:53.999 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster sgkydb cache info from db 2025-04-08 21:15:54.000 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster sgkydb cache info from db success. 2025-04-08 21:15:54.019 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000009, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:15:54.355 INFO 26087 --- [prometheus-minute-schedule-1,73816d22ce6ddd4f,d7abac2c46598827] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:54.414 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000009 2025-04-08 21:15:54.601 INFO 26087 --- [ocp-async-5,dee506b25975e0b1,78e8ffa1cdbba7eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:54.601 WARN 26087 --- [ocp-async-7,d67f896a7041c40d,5f48dca284c5518a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:56.299 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:56.299 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:15:56.537 INFO 26087 --- [ocp-schedules-30,f382a6ada769ca13,fbc9f7790a862f69] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:15:56.537 INFO 26087 --- [ocp-schedules-29,24fdf14b2086a614,6106008204b03b01] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:15:56.549 INFO 26087 --- [ocp-async-5,f6391ac701605fa5,065533fb552e5c2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:15:56.601 INFO 26087 --- [ocp-async-1,7f392c4f3e35d783,255afad7e3e2bad1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:56.601 WARN 26087 --- [ocp-async-9,fbba8726e003dd36,d1e1ced2b0446392] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:15:56.623087+08:00, lastProcessTime=2025-04-08T21:15:46.623382+08:00 2025-04-08 21:15:56.644 INFO 26087 --- [ocp-schedules-12,130d5aa963651b0d,0851a9dd2ab96c00] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:15:56.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:15:56.623087+08:00 2025-04-08 21:15:57.156 INFO 26087 --- [ocp-schedules-8,c6e18a28ea6259bd,03e99e3cad600855] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:15:57.164 INFO 26087 --- [ocp-schedules-8,c6e18a28ea6259bd,03e99e3cad600855] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:15:57.164 INFO 26087 --- [ocp-schedules-8,c6e18a28ea6259bd,03e99e3cad600855] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:15:57.355 INFO 26087 --- [prometheus-minute-schedule-1,66a0227d21fb128b,045aaab351078b77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118160, interval=60 2025-04-08 21:15:57.795 INFO 26087 --- [ocp-schedules-21,250caa4786a665e7,3d7b24fec7515747] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:15:57.797 INFO 26087 --- [ocp-schedules-21,250caa4786a665e7,3d7b24fec7515747] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:15:58.601 INFO 26087 --- [ocp-async-3,44b411d984dcd674,dcfa1050e4212d4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:15:58.601 WARN 26087 --- [ocp-async-8,f0945d9c85818d9f,70d5066682645e44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:15:59.210 INFO 26087 --- [ocp-schedules-25,d4c2013aa695ee1a,e134468a2c5febd3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:15:59.299 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:15:59.300 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:00.355 INFO 26087 --- [prometheus-minute-schedule-1,c24f120970cbda69,513f9d07a21ce897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:00.601 WARN 26087 --- [ocp-async-6,af2165f775ee2e41,9c6000ccb15d7225] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:00.601 INFO 26087 --- [ocp-async-7,ead6b0f839657461,b0ced68e98674b9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:02.300 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:02.301 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:02.601 WARN 26087 --- [ocp-async-10,078fd911038cac5a,dd215772c5b0c689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:02.601 INFO 26087 --- [ocp-async-5,aea01a905478f126,8e799a58aee7afe8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:03.355 INFO 26087 --- [prometheus-minute-schedule-1,53fe4a309c3001d0,740b7da43eb3a375] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:04.602 WARN 26087 --- [ocp-async-8,fdc2ec8cddc73e1d,2d7621ce65b6bda9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:04.602 INFO 26087 --- [ocp-async-2,b7ed817486456353,9ce17d08b5d8655b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:05.301 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:05.302 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:06.355 INFO 26087 --- [prometheus-minute-schedule-1,03b3db24471bb7fa,a79d32d6e10f7ab2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:06.355 INFO 26087 --- [prometheus-cache-monitor1,6284433c4b64a06a,1f54cabd587538a3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1496,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":426,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:16:06.355 INFO 26087 --- [prometheus-cache-monitor1,6284433c4b64a06a,1f54cabd587538a3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:16:06.380 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:12:06.36, end=2025-04-08T21:14:06.378} 2025-04-08 21:16:06.384 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:16:06.384 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:16:06.550 INFO 26087 --- [ocp-async-2,a16495e4f1aaaa06,199bc37fb49a3076] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:16:06.602 WARN 26087 --- [ocp-async-7,cc87260c8b11be7c,e6e5bb28db39896a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:06.602 INFO 26087 --- [ocp-async-4,3aa895490a61d1e8,9859b0bd7a868840] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:16:06.622975+08:00, lastProcessTime=2025-04-08T21:15:56.623087+08:00 2025-04-08 21:16:06.645 INFO 26087 --- [ocp-schedules-8,88199d0478d82c62,f5c9b2bd57069199] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:16:06.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:16:06.622975+08:00 2025-04-08 21:16:07.802 INFO 26087 --- [ocp-schedules-5,21e6139032415dba,66178c41bb1ab01f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:16:07.803 INFO 26087 --- [ocp-schedules-5,21e6139032415dba,66178c41bb1ab01f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:16:08.302 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:08.303 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:08.602 WARN 26087 --- [ocp-async-9,1ad45981ac0a4923,0fd626adfe6271ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:08.602 INFO 26087 --- [ocp-async-5,9ff2b984478676ab,bc3f91f04f555396] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:09.234 INFO 26087 --- [ocp-schedules-22,066a4f7af3c5613e,34b52822ad4fe8e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:16:09.355 INFO 26087 --- [prometheus-minute-schedule-1,856f0b99cd3a3916,b70d3c046a169aae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:10.001 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:16:10.002 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:16:10.602 WARN 26087 --- [ocp-async-8,a9833a7df2b01a5a,81baad1759504e6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:10.602 INFO 26087 --- [ocp-async-2,f8a28d2c9f22cbe9,773a2d6658fd4464] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:10.603 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:16:11.303 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:11.303 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:12.355 INFO 26087 --- [prometheus-minute-schedule-1,caa87ba0483c1b1a,b31ae4e809f0edce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:12.393 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118100, nodeCount=66, affectRows=66 2025-04-08 21:16:12.394 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118100, nodeCount=65, validCount=65 2025-04-08 21:16:12.401 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118100, nodeCount=141, affectRows=141 2025-04-08 21:16:12.402 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118100, nodeCount=140, validCount=140 2025-04-08 21:16:12.405 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118100, nodeCount=167, affectRows=167 2025-04-08 21:16:12.406 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118100, nodeCount=166, validCount=166 2025-04-08 21:16:12.602 INFO 26087 --- [ocp-async-7,271188294b7e405c,117d894e8d1a951a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:12.602 WARN 26087 --- [ocp-async-6,f6baee24ac7ef0e1,ca517295716f6219] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:14.304 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:14.304 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:14.602 WARN 26087 --- [ocp-async-9,97495d78eac25fc9,4364f57f58f9f951] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:14.602 INFO 26087 --- [ocp-async-10,f8a614e2523aa25b,46d55c9abed68d51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:15.355 INFO 26087 --- [prometheus-minute-schedule-1,d0086555c1286d91,cba2e965ec65228e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:16.517 INFO 26087 --- [ocp-schedules-14,598f6ba65836774f,ef5c83a5a8bab6b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:16:16.518 INFO 26087 --- [ocp-schedules-21,d02778a2efff0a4e,b2b6660339d3ac5a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:16:16.521 INFO 26087 --- [ocp-schedules-4,972c044b8ab48bf2,3542fd3b1f4f61cd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:16:16.521 INFO 26087 --- [ocp-schedules-17,3fadcae542234329,6f8365f4c6e453a9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:16:16.521 INFO 26087 --- [ocp-schedules-17,3fadcae542234329,6f8365f4c6e453a9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:16:16.523 INFO 26087 --- [ocp-schedules-12,6a518095ad18e479,9dcb600f5806524c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:16:16.523 INFO 26087 --- [ocp-schedules-12,6a518095ad18e479,9dcb600f5806524c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:16:16.524 INFO 26087 --- [ocp-schedules-20,5f8b8ddb254cb865,7da4f2a7d124b188] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:16:16.524 INFO 26087 --- [ocp-schedules-20,5f8b8ddb254cb865,7da4f2a7d124b188] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:16:16.524 INFO 26087 --- [ocp-schedules-26,095846e0c2f9aa89,a278e35d66e33e5f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:16:16.525 INFO 26087 --- [ocp-schedules-27,fdb1294f5c98b90d,29dedf81a4c13b2e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:16:16.525 INFO 26087 --- [ocp-schedules-21,d02778a2efff0a4e,b2b6660339d3ac5a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:16:16.525 INFO 26087 --- [ocp-schedules-26,095846e0c2f9aa89,a278e35d66e33e5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:16.525 INFO 26087 --- [ocp-schedules-27,fdb1294f5c98b90d,29dedf81a4c13b2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:16.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"sgkydb","ObClusterId":1744016068,"ObRegion":"sgkydb","ObRegionId":1744016068,"ReadonlyRsList":[],"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1744118153625000}, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:16:16.546 INFO 26087 --- [ocp-schedules-14,598f6ba65836774f,ef5c83a5a8bab6b1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:16:16.550 INFO 26087 --- [ocp-async-8,044a15cf3c097453,82bf3f9c40a14d5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:16:16.550 INFO 26087 --- [ocp-schedules-23,648702ec45b45997,9cc73be416d32a6b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:16:16.559 INFO 26087 --- [ocp-schedules-25,6bf539d82eddfad3,6f55d047ed510210] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:16:16.559 INFO 26087 --- [ocp-schedules-25,6bf539d82eddfad3,6f55d047ed510210] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:16:16.587 INFO 26087 --- [ocp-schedules-2,3d8f7aa72a56254d,91311180fcf5d080] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:16:16.602 INFO 26087 --- [ocp-schedules-28,9f83092936a2f066,0bb53d08e3453df5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:16:16.602 WARN 26087 --- [ocp-async-2,61c534491b03c63f,5a8950871198ccdf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:16.602 INFO 26087 --- [ocp-async-3,a6b14948b5b7e5cf,4e84ead2eb9ca47f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:16.615 INFO 26087 --- [ocp-schedules-30,0b877fbbb9ca28a2,e0a9267f4f6fe088] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:16:16.615 INFO 26087 --- [ocp-schedules-30,0b877fbbb9ca28a2,e0a9267f4f6fe088] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:16:16.620 INFO 26087 --- [ocp-schedules-16,08eda80789c0e03c,7bb05dee7ce41161] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:16:16.620 INFO 26087 --- [ocp-schedules-16,8fa6fb0f5930d720,a09143f452f359bb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:16:16.620 INFO 26087 --- [ocp-schedules-16,8fa6fb0f5930d720,a09143f452f359bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:16:16.623624+08:00, lastProcessTime=2025-04-08T21:16:06.622975+08:00 2025-04-08 21:16:16.627 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:16:16.627 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"sgkydb","ObClusterId":1744016068,"ObRegion":"sgkydb","ObRegionId":1744016068,"ReadonlyRsList":[],"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1744118153625000}, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:16:16.636 INFO 26087 --- [ocp-schedules-12,3e1fdb9f1bb7d188,9710393d03e70321] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:16:16.636 INFO 26087 --- [ocp-schedules-12,aea79a20bc4c636d,4a68fc2398743d10] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:16:16.636 INFO 26087 --- [ocp-schedules-12,aea79a20bc4c636d,4a68fc2398743d10] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:16:16.637 INFO 26087 --- [ocp-schedules-21,3b84747fab846c59,fc32a25b6c66be8b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:16:16.637+08:00, clusters=[1000002] 2025-04-08 21:16:16.638 INFO 26087 --- [ocp-schedules-21,3b84747fab846c59,fc32a25b6c66be8b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster sgkydb, trans stat result is empty. 2025-04-08 21:16:16.643 INFO 26087 --- [ocp-schedules-3,529a2ebd9db394bf,4a89417f59b70ee7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:16:16.643+08:00, clusters=[1000002] 2025-04-08 21:16:16.644 INFO 26087 --- [ocp-schedules-3,529a2ebd9db394bf,4a89417f59b70ee7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster sgkydb for slowSqlStat, sql count is 0. 2025-04-08 21:16:16.646 INFO 26087 --- [ocp-schedules-7,a1d181be7e5f77ec,a15348fdadc87ebc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:16:16.658 INFO 26087 --- [ocp-schedules-11,80a96787dff035ec,1988e0dc8139d09d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:16:16.658 INFO 26087 --- [ocp-schedules-11,80a96787dff035ec,1988e0dc8139d09d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:16:16.663 INFO 26087 --- [ocp-schedules-11,80a96787dff035ec,1988e0dc8139d09d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:16:16.663 INFO 26087 --- [ocp-schedules-11,80a96787dff035ec,1988e0dc8139d09d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:16:16.667 INFO 26087 --- [ocp-schedules-9,1ff81fc76fe1b1de,9461e359d9ae2e06] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:16:16.671 INFO 26087 --- [ocp-schedules-9,1ff81fc76fe1b1de,9461e359d9ae2e06] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:16:16.671 INFO 26087 --- [ocp-schedules-9,1ff81fc76fe1b1de,9461e359d9ae2e06] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:16:16.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:16:16.623624+08:00 2025-04-08 21:16:16.747 INFO 26087 --- [ocp-schedules-24,77723e7154f04e04,9e144801472e96f9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:16:16.797 INFO 26087 --- [ocp-schedules-25,4e20f9fb5dc93838,3542c21a711118b9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:16:16.798 INFO 26087 --- [ocp-schedules-25,4e20f9fb5dc93838,3542c21a711118b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:16.813 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:16:16.813 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:16:16.814 INFO 26087 --- [ocp-schedules-1,a3cb97f6d448737d,1bf391328df2caad] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:16:16.815 INFO 26087 --- [ocp-schedules-1,a3cb97f6d448737d,1bf391328df2caad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:16.827 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:16:16.827 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:16:16.830 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:16:16.830 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208338840276992 2025-04-08 21:16:16.832 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=766209785803853824 2025-04-08 21:16:16.832 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:16:16.832 INFO 26087 --- [ocp-schedules-10,ea1a758bd72b009b,23ef3a7fe2392dd8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:16:16.836 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [sgkydb, sys] 2025-04-08 21:16:16.837 ERROR 26087 --- [ocp-schedules-1,a3cb97f6d448737d,1bf391328df2caad] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=sgkydb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:16:16.837 INFO 26087 --- [ocp-schedules-1,a3cb97f6d448737d,1bf391328df2caad] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:16:17.165 INFO 26087 --- [ocp-schedules-6,e6f4975177055fb6,081c9ecbe1be83d9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:16:17.171 INFO 26087 --- [ocp-schedules-6,e6f4975177055fb6,081c9ecbe1be83d9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:16:17.171 INFO 26087 --- [ocp-schedules-6,e6f4975177055fb6,081c9ecbe1be83d9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:16:17.304 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:17.305 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:17.528 INFO 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.64, 192.168.1.65, 192.168.1.62, 192.168.1.63] 2025-04-08 21:16:17.528 INFO 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000004 2025-04-08 21:16:17.528 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000002 2025-04-08 21:16:17.528 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2025-04-08 21:16:17.528 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000005 2025-04-08 21:16:17.530 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:16:17.530 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:16:17.530 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:16:17.530 INFO 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:16:17.532 WARN 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:16:17.532 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:16:17.532 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:16:17.532 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:16:17.532 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:16:17.532 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:16:17.532 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:16:17.532 INFO 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:16:17.533 WARN 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:16:17.533 WARN 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:16:17.533 WARN 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:16:17.533 WARN 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:16:17.536 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.65:62889/api/v1/log/alarms, params:null 2025-04-08 21:16:17.536 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.64:62889/api/v1/log/alarms, params:null 2025-04-08 21:16:17.537 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.63:62889/api/v1/log/alarms, params:null 2025-04-08 21:16:17.537 INFO 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.62:62889/api/v1/log/alarms, params:null 2025-04-08 21:16:17.540 WARN 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:16:17.542 ERROR 26087 --- [ocp-schedules-21,44f37411001f9171,ac087cb124580b6e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 21 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.63:62889 [/192.168.1.63] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 35 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 44 common frames omitted 2025-04-08 21:16:17.812 INFO 26087 --- [ocp-schedules-11,d16b5cf8556a2c2d,ef05f36ae92447d9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:16:17.813 INFO 26087 --- [ocp-schedules-11,d16b5cf8556a2c2d,ef05f36ae92447d9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:16:18.355 INFO 26087 --- [prometheus-minute-schedule-1,cc9e6ad2064a8f9f,d3e9c55d97749395] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:18.602 WARN 26087 --- [ocp-async-4,6de8d45436e17e63,722ad59595059c45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:18.603 INFO 26087 --- [ocp-async-7,47a3b64048350574,7dc982afd5c5044a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:19.253 INFO 26087 --- [ocp-schedules-9,8aa8487d02ea5bbd,935d06bc079ad330] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:16:20.305 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:20.306 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:16:20.603 WARN 26087 --- [ocp-async-7,30c79bd03c4934ba,49e7ecac6e276060] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:20.603 INFO 26087 --- [ocp-async-10,1f58cc831b3756fc,85a02a8740b48b1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:21.355 INFO 26087 --- [prometheus-minute-schedule-1,a49b73d477951af2,2561c53863f2dd2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:22.603 WARN 26087 --- [ocp-async-1,4761a4f646c06de0,9611e1e7e014f7b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:22.603 INFO 26087 --- [ocp-async-2,d86ed5ba1911e6cf,3e3b0afd8865d2b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:23.306 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:23.306 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:24.355 INFO 26087 --- [prometheus-minute-schedule-1,fa03311cc1b2ecb5,40b6eb3672d1992c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:24.603 WARN 26087 --- [ocp-async-3,a81f1eaecaeef0bc,02f07b76b18a8053] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:24.603 INFO 26087 --- [ocp-async-4,df1e9c0eceb59ba3,20a6204c309cb72e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:26.307 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:26.307 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:26.526 INFO 26087 --- [ocp-schedules-27,fdb1294f5c98b90d,29dedf81a4c13b2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:16:26.526 INFO 26087 --- [ocp-schedules-26,095846e0c2f9aa89,a278e35d66e33e5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:16:26.550 INFO 26087 --- [ocp-async-7,21ddd41393483e45,c53068a208356f73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:16:26.603 WARN 26087 --- [ocp-async-1,cc2848c6d07cb11c,ab431727d58532b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:26.603 INFO 26087 --- [ocp-async-5,7b984409af943144,0a2fc5f20cf1225e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:16:26.623803+08:00, lastProcessTime=2025-04-08T21:16:16.623624+08:00 2025-04-08 21:16:26.647 INFO 26087 --- [ocp-schedules-13,d2157b4ec5b6882e,700b820b5a398501] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:16:26.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:16:26.623803+08:00 2025-04-08 21:16:27.355 INFO 26087 --- [prometheus-minute-schedule-1,1783c53e6a333fcb,2b33b245816e2208] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:27.816 INFO 26087 --- [ocp-schedules-19,14b7619e4a37ecb1,0cab7911345e4d1d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:16:27.817 INFO 26087 --- [ocp-schedules-19,14b7619e4a37ecb1,0cab7911345e4d1d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:16:28.603 WARN 26087 --- [ocp-async-3,ba9ab846bb072049,5ad320fed5532548] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:28.603 INFO 26087 --- [ocp-async-6,971d821e153a6614,c8bc75bf3a5e4dfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:29.283 INFO 26087 --- [ocp-schedules-15,b2dfee8364090b20,77641c01f78d454d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:16:29.307 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:29.308 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:30.355 INFO 26087 --- [prometheus-minute-schedule-1,a69ca312aeb83d44,4521795c6b8a271d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:30.603 WARN 26087 --- [ocp-async-2,7649600a4eed4606,6d949875a3347c39] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:30.603 INFO 26087 --- [ocp-async-9,23259e512d6aafa8,cbd4f4adb0ef55d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:32.308 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:32.308 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:32.604 WARN 26087 --- [ocp-async-9,9a458c10a98e4106,ce909bc8fc92c159] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:32.604 INFO 26087 --- [ocp-async-1,b205d1d17933978c,18de2144c6b50279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:33.355 INFO 26087 --- [prometheus-minute-schedule-1,5fcba7b987eb2843,ac12986f89304cc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:34.604 WARN 26087 --- [ocp-async-1,61d9d2c2dc551fd6,81a8c3263e4472b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:34.604 INFO 26087 --- [ocp-async-3,96f5a2eaa7f06269,f238d8d73e110dc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:35.309 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:35.309 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:36.355 INFO 26087 --- [prometheus-minute-schedule-1,2be8ec62883b7eff,3ba689616ad64dfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:36.550 INFO 26087 --- [ocp-async-6,a3058b8b0e58549e,ee4496e8c72ae1d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:16:36.604 WARN 26087 --- [ocp-async-4,88652541839a1cb0,8e09bdc76832516a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:36.604 INFO 26087 --- [ocp-async-2,0b7acae292e29cc4,edd3899f5dc70900] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:16:36.623524+08:00, lastProcessTime=2025-04-08T21:16:26.623803+08:00 2025-04-08 21:16:36.649 INFO 26087 --- [ocp-schedules-10,7af62190e90e86e9,684675704e1dab6d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:16:36.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:16:36.623524+08:00 2025-04-08 21:16:37.172 INFO 26087 --- [ocp-schedules-4,9134d7bc980b54bd,17077513057d05ae] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:16:37.181 INFO 26087 --- [ocp-schedules-4,9134d7bc980b54bd,17077513057d05ae] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:16:37.181 INFO 26087 --- [ocp-schedules-4,9134d7bc980b54bd,17077513057d05ae] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:16:37.824 INFO 26087 --- [ocp-schedules-13,4ec2b324b556711c,f639b629cb028d2f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:16:37.826 INFO 26087 --- [ocp-schedules-13,4ec2b324b556711c,f639b629cb028d2f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:16:37.851 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/1000007/rollback, client 10.11.12.102, traceId 0246de30c1af1205] 2025-04-08 21:16:37.854 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/1000007/rollback?taskInstanceId=1000007, client=10.11.12.102, traceId=0246de30c1af1205, method=NoDataResponse com.oceanbase.ocp.server.common.controller.TaskController.rollbackTask(Long), args=1000007,] 2025-04-08 21:16:37.866 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 1000061, current state: FAILED, new state: READY 2025-04-08 21:16:37.866 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 1000091, current state: FAILED, new state: READY 2025-04-08 21:16:37.866 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 1000115, current state: FAILED, new state: READY 2025-04-08 21:16:37.866 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 1000077, current state: FAILED, new state: READY 2025-04-08 21:16:37.916 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,0246de30c1af1205,c743abc04eb222a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/1000007/rollback client=10.11.12.102, traceId=0246de30c1af1205, duration=67 ms] 2025-04-08 21:16:38.149 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,d624c3e61023dede,4d65dcf7ec3c5a65] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=d624c3e61023dede, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:16:38.150 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,4a805cc3b7a925f2,c35be7b08489054a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=4a805cc3b7a925f2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:38.150 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,d624c3e61023dede,4d65dcf7ec3c5a65] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=d624c3e61023dede, duration=4 ms] 2025-04-08 21:16:38.166 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,4a805cc3b7a925f2,c35be7b08489054a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=4a805cc3b7a925f2, duration=20 ms] 2025-04-08 21:16:38.309 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:38.310 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:38.524 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000061, original=READY 2025-04-08 21:16:38.528 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000061, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:38.575 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000077, original=READY 2025-04-08 21:16:38.576 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000077, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:38.582 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000091, original=READY 2025-04-08 21:16:38.584 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000091, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:38.604 WARN 26087 --- [ocp-async-5,fbfc0f44e6647b09,7231b02949898df2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:38.604 INFO 26087 --- [ocp-async-9,0a1e30cd52452378,ffbd8eefb67c566a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:38.606 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000115, original=READY 2025-04-08 21:16:38.608 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,98342263af81a85d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000115, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:38.584 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:38.588 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:38.612 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000061, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:38.523+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000061}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:38.628 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000077, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:38.574+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000077}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:38.630 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:38.634 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:16:38.635 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:38.636 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:16:38.640 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000091, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:38.582+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000091}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:38.642 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:16:38.652 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000115, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:38.606+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000115}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:38.658 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:16:38.662 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:16:38.666 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:16:38.676 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback started 2025-04-08 21:16:38.678 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:16:38.683 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:16:38.725 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:16:38.693 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:16:38.704 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:16:38.717 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:16:38.736 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:16:38.769 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:16:38.780 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.786 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:16:38.790 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:16:38.791 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.794 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.802 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:16:38.806 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.806 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:16:38.811 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.827 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.846 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,b8d6446c6c6930de,ba2e0853d1e8c046] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments?taskInstanceId=1000007&subtaskInstanceId=1000061, client=10.11.12.102, traceId=b8d6446c6c6930de, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000007,1000061,] 2025-04-08 21:16:38.848 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,b8d6446c6c6930de,ba2e0853d1e8c046] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments client=10.11.12.102, traceId=b8d6446c6c6930de, duration=6 ms] 2025-04-08 21:16:38.839 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.847 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:16:38.852 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:16:38.869 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:16:38.879 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:16:38.902 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.906 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:16:38.910 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.911 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.913 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:16:38.919 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.919 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.924 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.931 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.934 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.937 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.939 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:16:38.942 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:16:38.965 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.975 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:16:38.983 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:16:38.991 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:38.993 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.006 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.012 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.026 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:16:39.030 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.031 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:16:39.035 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:16:39.045 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:16:39.073 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.085 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.099 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.103 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.105 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.112 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.114 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:16:39.118 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.124 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:16:39.141 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63 2025-04-08 21:16:39.176 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:16:39.185 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.195 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.200 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.205 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62 2025-04-08 21:16:39.209 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.230 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.237 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65 2025-04-08 21:16:39.241 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64 2025-04-08 21:16:39.243 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:39.254 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.283 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:16:39.304 INFO 26087 --- [ocp-schedules-12,b0576007cb2113fa,a55f3f245f00b838] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:16:39.284 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.306 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:39.301 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.355 INFO 26087 --- [prometheus-minute-schedule-1,536749c4731d9267,da79732efedbd305] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:39.301 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.308 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.313 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:16:39.331 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.371 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63 2025-04-08 21:16:39.374 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.383 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:39.396 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.405 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:39.412 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:16:39.428 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62 2025-04-08 21:16:39.443 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:16:39.463 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.499 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.504 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65 2025-04-08 21:16:39.539 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64 2025-04-08 21:16:39.777 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,99d8c7bf2a2b61bd,b316e18197a68c6e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=99d8c7bf2a2b61bd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:39.793 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,99d8c7bf2a2b61bd,b316e18197a68c6e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=99d8c7bf2a2b61bd, duration=20 ms] 2025-04-08 21:16:39.739 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.807 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.823 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:16:39.838 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.867 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.917 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:39.923 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.924 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.936 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.938 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:16:39.962 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:16:39.964 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:39.968 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:16:40.001 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:16:40.001 INFO 26087 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:16:39.978 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.63 2025-04-08 21:16:39.980 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:39.982 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:16:39.985 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:40.019 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.023 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:16:40.025 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:16:40.046 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:16:40.080 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.63 2025-04-08 21:16:40.084 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.62 2025-04-08 21:16:40.089 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:16:40.092 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.65 2025-04-08 21:16:40.112 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.64 2025-04-08 21:16:40.124 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.134 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.62 2025-04-08 21:16:40.135 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.139 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.182 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.65 2025-04-08 21:16:40.185 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.64 2025-04-08 21:16:40.273 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,9ea2576a108ee244,2959806a6d378579] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments?taskInstanceId=1000007&subtaskInstanceId=1000061, client=10.11.12.102, traceId=9ea2576a108ee244, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000007,1000061,] 2025-04-08 21:16:40.275 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,9ea2576a108ee244,2959806a6d378579] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments client=10.11.12.102, traceId=9ea2576a108ee244, duration=6 ms] 2025-04-08 21:16:40.456 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.63, result: SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.471 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.63,result:SshResult(host=192.168.1.63, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.475 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:16:40.477 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:16:40.478 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:16:40.492 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,dd33765914059ce2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000061, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:40.531 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.62, result: SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.533 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.62,result:SshResult(host=192.168.1.62, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.534 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:16:40.536 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:16:40.538 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.65, result: SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.538 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:16:40.539 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.65,result:SshResult(host=192.168.1.65, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.540 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.64, result: SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.543 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:16:40.545 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.64,result:SshResult(host=192.168.1.64, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:16:40.546 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,761982e96eaf8afb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000115, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:40.547 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:16:40.549 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:16:40.555 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:16:40.561 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:16:40.568 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] rollback completed 2025-04-08 21:16:40.569 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,0fcb075ad113ed71] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000077, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:40.573 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000091, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:40.574 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,8a89b42565ecdf2e] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000104] 2025-04-08 21:16:40.604 WARN 26087 --- [ocp-async-3,868c6a9c4e749305,f070e3e116499c14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:40.604 INFO 26087 --- [ocp-async-10,da6d41e02e292377,3c30b1f59d6948c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:40.631 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,9fb642260f40f781] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000104, original=READY 2025-04-08 21:16:40.633 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,9fb642260f40f781] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000104, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:40.679 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c693bc4e06b13640] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:40.684 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,c693bc4e06b13640] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000104, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:40.631+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000104}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:40.696 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c693bc4e06b13640] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000104, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:40.700 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c693bc4e06b13640] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000071, 1000045, 1000112, 1000087] 2025-04-08 21:16:41.261 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,13a4ad598e5a73ad,1ba399f6c51427ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=13a4ad598e5a73ad, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:41.277 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,13a4ad598e5a73ad,1ba399f6c51427ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=13a4ad598e5a73ad, duration=20 ms] 2025-04-08 21:16:41.310 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:41.311 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:41.624 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,2b77cfa4d5323317,ffc6dbbfe5ff9db4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments?taskInstanceId=1000007&subtaskInstanceId=1000061, client=10.11.12.102, traceId=2b77cfa4d5323317, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000007,1000061,] 2025-04-08 21:16:41.626 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,2b77cfa4d5323317,ffc6dbbfe5ff9db4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007/subtasks/1000061/logSegments client=10.11.12.102, traceId=2b77cfa4d5323317, duration=6 ms] 2025-04-08 21:16:41.647 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000045, original=READY 2025-04-08 21:16:41.648 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000045, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:41.726 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000071, original=READY 2025-04-08 21:16:41.727 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000071, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:41.732 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000087, original=READY 2025-04-08 21:16:41.733 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000087, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:41.732 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,de36b1d7c00f29d5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:41.766 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000112, original=READY 2025-04-08 21:16:41.768 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,7ff4e31eaf010908] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000112, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:41.737 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,99ce4477331bc0b0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:41.764 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,de36b1d7c00f29d5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000045, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:41.647+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000045}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:41.772 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,99ce4477331bc0b0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000071, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:41.725+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000071}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:41.779 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1c79d4683c97cf77] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:41.784 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,d861f064ddc4d995] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:41.785 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,de36b1d7c00f29d5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000045, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:41.787 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1c79d4683c97cf77] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000087, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:41.732+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000087}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:41.792 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,99ce4477331bc0b0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000071, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:41.794 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,d861f064ddc4d995] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000112, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:41.766+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000112}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:41.811 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1c79d4683c97cf77] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000087, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:41.812 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,d861f064ddc4d995] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000112, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:41.813 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,d861f064ddc4d995] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000080] 2025-04-08 21:16:42.355 INFO 26087 --- [prometheus-minute-schedule-1,35891f5d9eafbc96,44d72f4a3c2710f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:42.603 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,7a294c19fefff55f,94993fc3457aaa75] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=7a294c19fefff55f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:42.604 INFO 26087 --- [ocp-async-4,13815cb38242cb5e,88ee86ecd08bb80f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:42.604 WARN 26087 --- [ocp-async-6,f035bb15210b30bc,dbc0e0263e2e915b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:42.618 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,7a294c19fefff55f,94993fc3457aaa75] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=7a294c19fefff55f, duration=19 ms] 2025-04-08 21:16:42.781 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,83fde87b5f21b2c8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000080, original=READY 2025-04-08 21:16:42.782 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,83fde87b5f21b2c8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000080, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:42.871 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ffdb7d415547d95] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:42.881 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ffdb7d415547d95] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000080, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:42.780+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000080}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:42.907 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ffdb7d415547d95] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000080, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:42.909 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,5ffdb7d415547d95] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000046, 1000073, 1000098, 1000088] 2025-04-08 21:16:43.826 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000046, original=READY 2025-04-08 21:16:43.828 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000046, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:43.835 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000073, original=READY 2025-04-08 21:16:43.836 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000073, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:43.850 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000088, original=READY 2025-04-08 21:16:43.851 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000088, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:43.851 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,688a64db1700f37c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:43.879 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000098, original=READY 2025-04-08 21:16:43.880 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,af716e3b1a52d293] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000098, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:43.876 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d83931ad3d07059e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:43.879 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,688a64db1700f37c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000046, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:43.826+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000046}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:43.882 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,d83931ad3d07059e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000073, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:43.835+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000073}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:43.884 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,688a64db1700f37c] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 21:16:43.885 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,3e259bbe762183e6] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:43.892 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,d267a4aebfdab09c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:43.908 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d83931ad3d07059e] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 21:16:43.916 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,3e259bbe762183e6] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000088, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:43.850+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000088}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:43.917 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,688a64db1700f37c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000046, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:43.919 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,d267a4aebfdab09c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000098, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:43.879+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000098}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:43.923 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,3e259bbe762183e6] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 21:16:43.924 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,d83931ad3d07059e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000073, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:43.931 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,48878952f67f3510,99e2faa2de6663e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=48878952f67f3510, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:43.924 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,d267a4aebfdab09c] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] no need to rollback 2025-04-08 21:16:43.940 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,3e259bbe762183e6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000088, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:43.941 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,d267a4aebfdab09c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000098, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:43.945 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,48878952f67f3510,99e2faa2de6663e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=48878952f67f3510, duration=17 ms] 2025-04-08 21:16:44.311 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:44.312 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:44.604 WARN 26087 --- [ocp-async-5,14202eaa2fae35ef,5c9f429c1dfd4504] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:44.604 INFO 26087 --- [ocp-async-8,432976c5b9cb233c,b68faf47ee0adfd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:45.068 INFO 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.AbstractTaskWorker : Set rolling back subtask to ready, subtaskIds=[1000050] 2025-04-08 21:16:45.282 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,7d8ae317bb0874f4,ccd704890011b8a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=7d8ae317bb0874f4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:45.297 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,7d8ae317bb0874f4,ccd704890011b8a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=7d8ae317bb0874f4, duration=19 ms] 2025-04-08 21:16:45.355 INFO 26087 --- [prometheus-minute-schedule-1,04aae9e9c1dd46a6,46556a58c9c8f8d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:45.888 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,38e5d628fafc73f6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000050, original=READY 2025-04-08 21:16:45.890 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,38e5d628fafc73f6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000050, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:45.914 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,76cd84095205085f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:45.917 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,76cd84095205085f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000050, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:45.888+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000050}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:45.927 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,76cd84095205085f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000050, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:45.928 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,76cd84095205085f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000049, 1000055, 1000089, 1000099] 2025-04-08 21:16:46.521 INFO 26087 --- [ocp-schedules-10,4e95a58ea0d343b3,497e2bda2df044b0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:16:46.521 INFO 26087 --- [ocp-schedules-26,018996adda81c969,da1d76a8e89b07e2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:16:46.521 INFO 26087 --- [ocp-schedules-26,018996adda81c969,da1d76a8e89b07e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:16:46.523 INFO 26087 --- [ocp-schedules-3,e0905a4abbf0b4fa,b0d18cc9e12708fb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:16:46.523 INFO 26087 --- [ocp-schedules-3,e0905a4abbf0b4fa,b0d18cc9e12708fb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:16:46.523 INFO 26087 --- [ocp-schedules-7,fea214613b24e555,6ef24dfbecb8c2ba] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:16:46.523 INFO 26087 --- [ocp-schedules-7,fea214613b24e555,6ef24dfbecb8c2ba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:16:46.524 INFO 26087 --- [ocp-schedules-19,6c976c12f21a7710,d7c7c32b7163c68a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:16:46.524 INFO 26087 --- [ocp-schedules-1,d4a94d0b19597f64,ad04bdd66d33854d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:16:46.525 INFO 26087 --- [ocp-schedules-19,6c976c12f21a7710,d7c7c32b7163c68a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:46.525 INFO 26087 --- [ocp-schedules-1,d4a94d0b19597f64,ad04bdd66d33854d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:46.525 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"sgkydb","ObClusterId":1744016068,"ObRegion":"sgkydb","ObRegionId":1744016068,"ReadonlyRsList":[],"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1744118153625000}, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:16:46.530 INFO 26087 --- [ocp-schedules-24,3a9946571ea30eaa,6f40430462ebb939] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:16:46.550 INFO 26087 --- [ocp-async-1,e58045cfff6741fd,b2b4b60c885e3ffd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:16:46.566 INFO 26087 --- [ocp-schedules-21,5acad666ee602080,5bb2fa9fe34e2496] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:16:46.567 INFO 26087 --- [ocp-schedules-21,5acad666ee602080,5bb2fa9fe34e2496] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:46.567 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:16:46.604 INFO 26087 --- [ocp-schedules-6,dfd11169dcbf1c4c,dafe1ed2a7491ee9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:16:46.604 WARN 26087 --- [ocp-async-3,1536476a265c53ae,95e31873779df2be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:46.604 INFO 26087 --- [ocp-async-10,2a6c2e8e6436f4b6,d586eaf3bfba7bb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:16:46.623802+08:00, lastProcessTime=2025-04-08T21:16:36.623524+08:00 2025-04-08 21:16:46.627 INFO 26087 --- [ocp-schedules-16,8fa6fb0f5930d720,a09143f452f359bb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:16:46.627 INFO 26087 --- [ocp-schedules-16,20f8953d8434fdbc,06b519a7f7fc9510] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:16:46.628 INFO 26087 --- [ocp-schedules-16,20f8953d8434fdbc,06b519a7f7fc9510] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000002] 2025-04-08 21:16:46.634 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000002, status=TAKINGOVER 2025-04-08 21:16:46.634 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000002, name=sgkydb, obClusterId=1744016068, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"sgkydb","ObClusterId":1744016068,"ObRegion":"sgkydb","ObRegionId":1744016068,"ReadonlyRsList":[],"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1744118153625000}, createTime=2025-04-08T21:09:56+08:00, creator=admin, obVersion=4.3.5.1, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:16:46.650 INFO 26087 --- [ocp-schedules-23,3f4f20c6e0c2e333,b908971330508a39] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:16:46.679 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,9b3040ae2212b452,e5152a7f299ce3f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=9b3040ae2212b452, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:46.695 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,9b3040ae2212b452,e5152a7f299ce3f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=9b3040ae2212b452, duration=19 ms] 2025-04-08 21:16:46.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:16:46.623802+08:00 2025-04-08 21:16:46.904 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000049, original=READY 2025-04-08 21:16:46.906 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000049, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:46.909 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000055, original=READY 2025-04-08 21:16:46.911 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000055, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:46.918 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000089, original=READY 2025-04-08 21:16:46.919 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000089, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:46.918 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,58a84629bf457f8e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:46.929 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000099, original=READY 2025-04-08 21:16:46.931 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,02fb82636d124267] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000099, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:46.926 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,bd1ab4c78b9a8980] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:46.929 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,58a84629bf457f8e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000049, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:46.904+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000049}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:46.934 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,bd1ab4c78b9a8980] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000055, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:46.909+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000055}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:46.939 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,3e9f02abf7ddcde0] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:46.940 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,58a84629bf457f8e] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 21:16:46.943 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,bd1ab4c78b9a8980] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 21:16:46.945 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,92ff77a36a4ac34c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:46.951 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,3e9f02abf7ddcde0] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000089, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:46.917+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000089}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:46.981 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,58a84629bf457f8e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000049, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:46.982 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,bd1ab4c78b9a8980] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000055, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:46.983 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,92ff77a36a4ac34c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000099, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:46.929+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000099}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:46.984 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,3e9f02abf7ddcde0] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 21:16:46.987 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,92ff77a36a4ac34c] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : no need to rollback 2025-04-08 21:16:47.009 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,3e9f02abf7ddcde0] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000089, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:47.011 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,92ff77a36a4ac34c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000099, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:47.013 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,92ff77a36a4ac34c] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000092] 2025-04-08 21:16:47.312 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:47.312 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:47.829 INFO 26087 --- [ocp-schedules-2,4b527868022484f7,b0e76c0e6b23d6e2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:16:47.831 INFO 26087 --- [ocp-schedules-2,4b527868022484f7,b0e76c0e6b23d6e2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:16:47.940 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,b5677044c0fd15ec] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000092, original=READY 2025-04-08 21:16:47.941 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,b5677044c0fd15ec] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000092, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:47.987 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,39916bdcc869d9e8] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:47.990 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,39916bdcc869d9e8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000092, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:47.940+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000092}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:48.023 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,32714abf17b1dec3,735cbb84df329d36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=32714abf17b1dec3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:48.029 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,39916bdcc869d9e8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000092, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:48.031 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,39916bdcc869d9e8] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000103, 1000096, 1000081, 1000066] 2025-04-08 21:16:48.038 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,32714abf17b1dec3,735cbb84df329d36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=32714abf17b1dec3, duration=18 ms] 2025-04-08 21:16:48.355 INFO 26087 --- [prometheus-minute-schedule-1,5e356eb30d2377d3,240cab39a2e8e557] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:48.605 WARN 26087 --- [ocp-async-6,6d37e55cadce5841,93fbaf300d0d95cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:48.605 INFO 26087 --- [ocp-async-4,c3066d6c4dee738b,67d619e711fea3f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:48.973 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000066, original=READY 2025-04-08 21:16:48.974 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000066, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:48.983 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000081, original=READY 2025-04-08 21:16:48.984 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000081, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:48.991 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000096, original=READY 2025-04-08 21:16:48.992 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000096, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:48.992 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,85078c2008614be8] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:49.019 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000103, original=READY 2025-04-08 21:16:49.021 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,335202a44ee2fe07] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000103, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:49.014 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,b93f5a50ca1e0ea8] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:49.019 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,85078c2008614be8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000066, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:48.973+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000066}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:49.040 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,b93f5a50ca1e0ea8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000081, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:48.983+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000081}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:49.044 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,c3eb08456ef16c0f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:49.055 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9ba95710fc24d2d4] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:49.083 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,85078c2008614be8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000066, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:49.084 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,b93f5a50ca1e0ea8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000081, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:49.085 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,c3eb08456ef16c0f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000096, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:48.991+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000096}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:49.091 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,9ba95710fc24d2d4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000103, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:49.019+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000103}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:49.121 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,c3eb08456ef16c0f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000096, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:49.123 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9ba95710fc24d2d4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000103, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:49.124 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,9ba95710fc24d2d4] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000064] 2025-04-08 21:16:49.352 INFO 26087 --- [ocp-schedules-27,0c4f55e562467be7,247d60dc24815374] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:16:49.382 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,6596fde2ac628955,8ff3189d82cac124] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=6596fde2ac628955, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:49.401 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,6596fde2ac628955,8ff3189d82cac124] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=6596fde2ac628955, duration=22 ms] 2025-04-08 21:16:50.044 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,f864f652fb9e1938] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000064, original=READY 2025-04-08 21:16:50.046 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,f864f652fb9e1938] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000064, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:50.087 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,3b3d0069ecfa9c5f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:50.090 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,3b3d0069ecfa9c5f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000064, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:50.044+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000064}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:50.111 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,3b3d0069ecfa9c5f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000064, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:50.113 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,3b3d0069ecfa9c5f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000078, 1000063, 1000110, 1000101] 2025-04-08 21:16:50.312 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:50.313 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:50.605 WARN 26087 --- [ocp-async-2,b81004ffe9d3feb4,96f85bb5a1e72c7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:50.605 INFO 26087 --- [ocp-async-8,343b8bed637ccf71,ed12e487ce423a5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:50.724 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,6b825d37d6d115db,30ab4dbe01dc3045] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=6b825d37d6d115db, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:50.739 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,6b825d37d6d115db,30ab4dbe01dc3045] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=6b825d37d6d115db, duration=20 ms] 2025-04-08 21:16:51.052 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000063, original=READY 2025-04-08 21:16:51.053 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000063, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:51.085 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000078, original=READY 2025-04-08 21:16:51.087 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000078, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:51.093 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000101, original=READY 2025-04-08 21:16:51.094 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000101, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:51.093 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:51.114 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000110, original=READY 2025-04-08 21:16:51.116 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,fa85c84d4474bba9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000110, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:51.103 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:51.114 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000063, context=Context{parallelIdx=2, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:51.052+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000063}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:51.118 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000078, context=Context{parallelIdx=0, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:51.085+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000078}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:51.122 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:51.124 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=1000004 2025-04-08 21:16:51.128 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:51.135 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=1000002 2025-04-08 21:16:51.160 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000101, context=Context{parallelIdx=3, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:51.093+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000101}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:51.170 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000004, method=deleteHost 2025-04-08 21:16:51.171 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000110, context=Context{parallelIdx=1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:51.114+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000110}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:51.185 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000002, method=deleteHost 2025-04-08 21:16:51.186 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=1000005 2025-04-08 21:16:51.192 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.s.t.b.host.PreCreateHostCheckTask : Delete all host info, hostId=1000003 2025-04-08 21:16:51.196 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 1000004, please retry manually 2025-04-08 21:16:51.215 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000005, method=deleteHost 2025-04-08 21:16:51.219 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 1000002, please retry manually 2025-04-08 21:16:51.222 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000003, method=deleteHost 2025-04-08 21:16:51.233 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=1000004 2025-04-08 21:16:51.282 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=1000002 2025-04-08 21:16:51.286 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 1000005, please retry manually 2025-04-08 21:16:51.287 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000004 2025-04-08 21:16:51.293 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.host.manager.HostTaskManager : Failed to cancel install ocp-agent task on host 1000003, please retry manually 2025-04-08 21:16:51.301 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2025-04-08 21:16:51.313 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=1000005 2025-04-08 21:16:51.355 INFO 26087 --- [prometheus-minute-schedule-1,4ef05784660fb172,29de51c3f57c631e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:51.320 WARN 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000004 2025-04-08 21:16:51.326 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.agent.HostAgentServiceImpl : Deleting host agent and processes: hostId=1000003 2025-04-08 21:16:51.339 WARN 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000002 2025-04-08 21:16:51.352 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000005 2025-04-08 21:16:51.383 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 1000004 2025-04-08 21:16:51.386 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2025-04-08 21:16:51.390 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 1000002 2025-04-08 21:16:51.394 WARN 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000005 2025-04-08 21:16:51.421 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [1000004] 2025-04-08 21:16:51.424 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000003 2025-04-08 21:16:51.430 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [1000002] 2025-04-08 21:16:51.430 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 1000005 2025-04-08 21:16:51.433 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [1000004] 2025-04-08 21:16:51.438 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.agent.HostAgentServiceImpl : Deleting OCP Agent host 1000003 2025-04-08 21:16:51.484 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [1000002] 2025-04-08 21:16:51.486 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [1000005] 2025-04-08 21:16:51.486 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [1000004] 2025-04-08 21:16:51.492 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.s.c.CredentialServiceImpl : Deleting HOST credentials access by resource ids [1000003] 2025-04-08 21:16:51.497 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [1000002] 2025-04-08 21:16:51.507 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [1000005] 2025-04-08 21:16:51.511 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.s.c.CredentialServiceImpl : Success to delete HOST credentials access by resource ids [1000003] 2025-04-08 21:16:51.512 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [1000004] 2025-04-08 21:16:51.513 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [1000002] 2025-04-08 21:16:51.514 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [1000005] 2025-04-08 21:16:51.519 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.s.c.CredentialServiceImpl : Deleting AGENT credentials access by resource ids [1000003] 2025-04-08 21:16:51.523 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [1000004:192.168.1.62], from status [NEW] to [NEW] 2025-04-08 21:16:51.526 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [1000002:192.168.1.64], from status [NEW] to [NEW] 2025-04-08 21:16:51.529 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [1000005] 2025-04-08 21:16:51.533 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.s.c.CredentialServiceImpl : Success to delete AGENT credentials access by resource ids [1000003] 2025-04-08 21:16:51.569 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [1000005:192.168.1.63], from status [NEW] to [NEW] 2025-04-08 21:16:51.587 INFO 26087 --- [manual-subtask-executor14,2eb36642664461ac,b9dd1b0a4f6b9bb5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000078, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:51.584 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.host.manager.HostDeletingHelper : Force deleting host: [1000003:192.168.1.65], from status [NEW] to [NEW] 2025-04-08 21:16:51.586 INFO 26087 --- [manual-subtask-executor13,2eb36642664461ac,c729b642edc75278] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000063, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:51.599 INFO 26087 --- [manual-subtask-executor15,2eb36642664461ac,7394cea7298670ec] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000101, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:51.603 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000110, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:51.605 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,db2af0facf267dfa] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000047] 2025-04-08 21:16:52.052 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,88244d5ccb7e75bf,88ec680056ca737c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=88244d5ccb7e75bf, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:52.067 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,88244d5ccb7e75bf,88ec680056ca737c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=88244d5ccb7e75bf, duration=19 ms] 2025-04-08 21:16:52.123 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,859252af0acecc56] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=1000047, original=READY 2025-04-08 21:16:52.124 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,859252af0acecc56] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000047, operation:ROLLBACK, state: ROLLING_BACK 2025-04-08 21:16:52.142 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:16:52.151 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=1000047, context=Context{parallelIdx=-1, stringMap={cluster_version=4.3.5.1, cluster_name=sgkydb, target_server_status=RUNNING, ssh_port=22, service_name=sgkydb:1744016068, target_zone_status=RUNNING, task_instance_id=1000007, ob_connect_address=192.168.1.62:2881, task_operation=rollback, cluster_type=PRIMARY, service_version=4.3.5.1, host_credential_id=1, ob_cluster_id=1744016068, cluster_id=1000002, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/sgkydb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:16:52.123+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=1000047}, listMap={add_region_ids=[1000003], server_ids=[1000002, 1000003, 1000004, 1000005], add_idc_ids=[1000002], all_host_ids=[1000002, 1000003, 1000004, 1000005], add_host_ids=[1000002, 1000003, 1000004, 1000005], host_ids=[1000002, 1000003, 1000004, 1000005], zone_names=[zone2, zone1]}}, operation=ROLLBACK, executor=192.168.1.61 2025-04-08 21:16:52.170 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster metadata 2025-04-08 21:16:52.196 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.o.i.c.l.CredentialServiceListener : Detect cluster been deleted, begin to delete credentials of this cluster, clusterId=1000002 2025-04-08 21:16:52.205 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.c.CredentialServiceImpl : Deleting credential: 1000006 2025-04-08 21:16:52.214 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.c.CredentialServiceImpl : Delete credential 1000006 success 2025-04-08 21:16:52.228 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.o.i.c.l.CredentialServiceListener : Delete credentials success, clusterId=1000002 2025-04-08 21:16:52.233 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.o.i.c.l.ClusterServiceListener : detect cluster been deleted, begin to delete parameter change and compaction info, clusterId=1000002 2025-04-08 21:16:52.283 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000010, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:16:52.288 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.o.i.c.l.ClusterServiceListener : detect parameter change and compaction info finished, clusterId=1000002 2025-04-08 21:16:52.293 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000011, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["sgkydb"]} 2025-04-08 21:16:52.297 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000005] 2025-04-08 21:16:52.298 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000005 at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$5d1537be.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:16:52.304 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000003] 2025-04-08 21:16:52.307 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000003 at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$5d1537be.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:16:52.314 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000004] 2025-04-08 21:16:52.318 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000004 at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$5d1537be.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:16:52.331 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000002] 2025-04-08 21:16:52.333 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.common.lang.function.Safety : Failed to execute runnable. com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000002 at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.lambda$nullSafeGetHost$6(HostAgentServiceImpl.java:924) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.nullSafeGetHost(HostAgentServiceImpl.java:924) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getExporterAddresses(HostAgentServiceImpl.java:1060) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.offlineExporter(HostAgentServiceImpl.java:365) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.offlineExporter() at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$null$1(ClusterServiceListener.java:65) at com.oceanbase.ocp.common.lang.function.Safety.safeExecute(Safety.java:22) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.lambda$onDeleteClusterEvent$2(ClusterServiceListener.java:64) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.cluster.listener.ClusterServiceListener.onDeleteClusterEvent(ClusterServiceListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:344) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:229) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:166) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:429) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:386) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl.deleteClusterRelatedInfo(ClusterServiceImpl.java:1594) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$FastClassBySpringCGLIB$$d321c173.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterServiceImpl$$EnhancerBySpringCGLIB$$5d1537be.deleteClusterRelatedInfo() at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.deleteCluster(PrepareTakeOverClusterTask.java:67) at com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask.rollback(PrepareTakeOverClusterTask.java:48) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.rollback(JavaSubtaskRunner.java:84) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:38) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:16:52.338 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.o.internal.ObproxyServiceListener : detect cluster been deleted, begin to delete oblinks, obclusterName=sgkydb, obClusterId=1744016068 2025-04-08 21:16:52.371 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000012, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:16:52.376 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.b.i.l.BackupServiceListener : detect cluster been deleted, begin to delete backup strategy, clusterId=1000002 2025-04-08 21:16:52.388 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.b.i.l.BackupServiceListener : detect backup strategy finished, clusterId=1000002 2025-04-08 21:16:52.441 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.s.o.OrganizationResourceManager : delete resource for current organization,organizationId=10000000,resourceType=1000002, resourceId=1000002 2025-04-08 21:16:52.452 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete cluster finished 2025-04-08 21:16:52.486 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host started, hostIds=[1000002, 1000003, 1000004, 1000005] 2025-04-08 21:16:52.496 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000002, method=deleteHost 2025-04-08 21:16:52.514 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000002] 2025-04-08 21:16:52.522 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 1000002 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000002 2025-04-08 21:16:52.529 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000003, method=deleteHost 2025-04-08 21:16:52.550 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000003] 2025-04-08 21:16:52.571 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 1000003 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000003 2025-04-08 21:16:52.578 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000004, method=deleteHost 2025-04-08 21:16:52.590 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000004] 2025-04-08 21:16:52.605 WARN 26087 --- [ocp-async-9,2e9eb8f9aef955af,aa8024db1b4c5cad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:52.605 INFO 26087 --- [ocp-async-1,45ac32fb30fd6168,e1e8747e594ae87a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:52.613 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 1000004 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000004 2025-04-08 21:16:52.622 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=delete-host-1000005, method=deleteHost 2025-04-08 21:16:52.633 ERROR 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.common.not.found, and args [HostEntity, 1000005] 2025-04-08 21:16:52.637 WARN 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host 1000005 failed, message=[OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMMON_NOT_FOUND, args=HostEntity,1000005 2025-04-08 21:16:52.641 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : delete host finished 2025-04-08 21:16:52.651 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.host.service.IdcServiceImpl : Deleted idc: id=1000002 2025-04-08 21:16:52.666 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.host.service.RegionServiceImpl : Deleted region: 1000003 sys_region 2025-04-08 21:16:52.686 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.s.t.b.c.PrepareTakeOverClusterTask : rollback finished 2025-04-08 21:16:52.695 INFO 26087 --- [manual-subtask-executor16,2eb36642664461ac,1b0d90104f50e940] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000047, operation:ROLLBACK, state: PENDING 2025-04-08 21:16:53.061 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [sgkydb] cache 2025-04-08 21:16:53.061 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:16:53.061 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster sgkydb cache info from db 2025-04-08 21:16:53.062 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster sgkydb not found in config server 2025-04-08 21:16:53.081 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000010, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:16:53.081 INFO 26087 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Receive invalidate ob version event, clusters:[sgkydb] 2025-04-08 21:16:53.082 INFO 26087 --- [ocp-updater-0,,] c.o.o.o.i.c.l.ClusterServiceListener : Invalidate ob version success, clusters:[sgkydb] 2025-04-08 21:16:53.084 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000011, event=InvalidateObVersionEvent{name=invalidate-ob-version-cache, payload=["sgkydb"]} 2025-04-08 21:16:53.084 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [sgkydb] cache 2025-04-08 21:16:53.084 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:16:53.084 INFO 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster sgkydb cache info from db 2025-04-08 21:16:53.084 WARN 26087 --- [ForkJoinPool.commonPool-worker-15,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Ob cluster sgkydb not found in config server 2025-04-08 21:16:53.087 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000012, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["sgkydb"]} 2025-04-08 21:16:53.313 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:53.314 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:53.383 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,6e63649bc0521c08,593401a0b241678f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=6e63649bc0521c08, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:53.399 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,6e63649bc0521c08,593401a0b241678f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=6e63649bc0521c08, duration=20 ms] 2025-04-08 21:16:53.423 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000012 2025-04-08 21:16:53.424 INFO 26087 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000011 2025-04-08 21:16:53.424 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000010 2025-04-08 21:16:53.788 ERROR 26087 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : GetClusterOverview, clusterId:1000002, NotFoundException {} com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_CLUSTER_ID_NOT_FOUND, args=1000002 at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl.nullSafeGetClusterOverview(ClusterDaoManagerImpl.java:94) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$FastClassBySpringCGLIB$$1f10cbd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.cluster.ClusterDaoManagerImpl$$EnhancerBySpringCGLIB$$5edf77dc.nullSafeGetClusterOverview() at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:126) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:16:53.788 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:16:54.355 INFO 26087 --- [prometheus-minute-schedule-1,d9bdb1adcb2b516b,a8b3bdd08f93cbbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:54.605 WARN 26087 --- [ocp-async-6,c44f8a323a9d2329,e7cb799cc6047287] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:54.605 INFO 26087 --- [ocp-async-10,39e2323871160246,4b013fb75dcda79a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:54.701 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,fce378db7f73fdfe,9661efb6f8cf16bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000007?taskInstanceId=1000007, client=10.11.12.102, traceId=fce378db7f73fdfe, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000007,] 2025-04-08 21:16:54.715 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,fce378db7f73fdfe,9661efb6f8cf16bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000007 client=10.11.12.102, traceId=fce378db7f73fdfe, duration=18 ms] 2025-04-08 21:16:56.314 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:56.315 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:56.525 INFO 26087 --- [ocp-schedules-1,d4a94d0b19597f64,ad04bdd66d33854d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:16:56.525 INFO 26087 --- [ocp-schedules-19,6c976c12f21a7710,d7c7c32b7163c68a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:16:56.550 INFO 26087 --- [ocp-async-10,320c7062cec33bfb,144882291ec614c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:16:56.567 INFO 26087 --- [ocp-schedules-21,5acad666ee602080,5bb2fa9fe34e2496] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:16:56.605 WARN 26087 --- [ocp-async-5,f5d521c46e7101e7,ac70a3c6872d7613] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:56.605 INFO 26087 --- [ocp-async-8,ea8ab216037233b7,084958aa3793541c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:16:56.622527+08:00, lastProcessTime=2025-04-08T21:16:46.623802+08:00 2025-04-08 21:16:56.652 INFO 26087 --- [ocp-schedules-24,83386cbcbfd58fd5,4c3716befbce9ac2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:16:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-sgkydb 2025-04-08 21:16:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:16:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-sgkydb 2025-04-08 21:16:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:16:56.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:16:56.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:16:56.622527+08:00 2025-04-08 21:16:56.896 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,fb290ef2e92ed89c,bd27022ad9343fcb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=fb290ef2e92ed89c, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:16:56.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,b02f314d7971064a,0f642ed599897065] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=b02f314d7971064a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:16:56.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,6758229bbf02cb5c,2d85ff514eb5644c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=6758229bbf02cb5c, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:16:56.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,b497343b375a8581,ba51ca2a48712395] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A16%3A56%2B08%3A00&endTime=2025-04-08T21%3A16%3A56%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=b497343b375a8581, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:16:56+08:00,2025-04-08T21:16:56+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:16:56.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,fb290ef2e92ed89c,bd27022ad9343fcb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=fb290ef2e92ed89c, duration=3 ms] 2025-04-08 21:16:56.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,5929e15215b9c37f,7d1fac26dad9e39e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=5929e15215b9c37f, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:16:56.898 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,5929e15215b9c37f,7d1fac26dad9e39e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=5929e15215b9c37f, duration=3 ms] 2025-04-08 21:16:56.899 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,6758229bbf02cb5c,2d85ff514eb5644c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=6758229bbf02cb5c, duration=6 ms] 2025-04-08 21:16:56.900 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,b02f314d7971064a,0f642ed599897065] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=b02f314d7971064a, duration=7 ms] 2025-04-08 21:16:56.902 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,b497343b375a8581,ba51ca2a48712395] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:16:56.902 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,b497343b375a8581,ba51ca2a48712395] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114620 2025-04-08 21:16:56.904 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,b497343b375a8581,ba51ca2a48712395] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=b497343b375a8581, duration=10 ms] 2025-04-08 21:16:56.922 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,25b26b46f348645f,8a0685c101baf28d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=25b26b46f348645f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:16:56.927 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,25b26b46f348645f,8a0685c101baf28d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=25b26b46f348645f, duration=10 ms] 2025-04-08 21:16:57.182 INFO 26087 --- [ocp-schedules-6,b9dfb1dc2436b1aa,cc31cf9ae9473840] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:16:57.187 INFO 26087 --- [ocp-schedules-6,b9dfb1dc2436b1aa,cc31cf9ae9473840] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:16:57.187 INFO 26087 --- [ocp-schedules-6,b9dfb1dc2436b1aa,cc31cf9ae9473840] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:16:57.355 INFO 26087 --- [prometheus-minute-schedule-1,95e5f7268a9a7a81,01cc0cf364e727ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118220, interval=60 2025-04-08 21:16:57.834 INFO 26087 --- [ocp-schedules-12,edf0d52fef200413,8e4c65783dd6fc56] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:16:57.835 INFO 26087 --- [ocp-schedules-12,edf0d52fef200413,8e4c65783dd6fc56] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:16:57.877 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,6a77f8a887373848,9327971b1c95e152] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=6a77f8a887373848, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:16:57.880 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,6a77f8a887373848,9327971b1c95e152] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=6a77f8a887373848, duration=6 ms] 2025-04-08 21:16:58.605 WARN 26087 --- [ocp-async-3,ecaadeb17eb902c6,e66e8c13ffcc4260] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:16:58.605 INFO 26087 --- [ocp-async-1,66fcea0e72416642,e3850df2822be5a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:16:59.315 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:16:59.315 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:16:59.374 INFO 26087 --- [ocp-schedules-26,7f406e29cd47d502,36a13f6ab0453373] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:17:00.355 INFO 26087 --- [prometheus-minute-schedule-1,25c618cd07750f48,9371e89f45d4d489] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:00.605 WARN 26087 --- [ocp-async-7,7a5d48f45af1856c,925483f13f7eb192] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:00.606 INFO 26087 --- [ocp-async-4,577a91dd1ffcfbc8,123cff02700a6d11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:02.316 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:02.316 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:02.606 WARN 26087 --- [ocp-async-4,6d6aba7a0eda7b7b,0dd76b75f019dd10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:02.606 INFO 26087 --- [ocp-async-2,709fce1a24a8cbfd,70f906bb6da2021e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:03.355 INFO 26087 --- [prometheus-minute-schedule-1,fb1befba12e9eef4,ff4d55c8e7fb7e26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:04.606 WARN 26087 --- [ocp-async-3,e2f9572599965026,6aa0cd5eff1ec70d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:04.606 INFO 26087 --- [ocp-async-9,dacc1d501759b698,d32d158602ab634c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:05.316 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:05.317 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:06.355 INFO 26087 --- [prometheus-minute-schedule-1,8ce1fcb32c3272b9,4fedd0b7d5c99e41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:06.355 INFO 26087 --- [prometheus-cache-monitor1,0a767204ffbf393b,e62a6f24313b216d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1496,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":376,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":373} 2025-04-08 21:17:06.355 INFO 26087 --- [prometheus-cache-monitor1,0a767204ffbf393b,e62a6f24313b216d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:17:06.550 INFO 26087 --- [ocp-async-1,5c6fd6d441301037,855757b6adb81458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:17:06.606 WARN 26087 --- [ocp-async-6,aff8d67e8a917dbf,8cb6cd49fb6232e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:06.606 INFO 26087 --- [ocp-async-10,39e291c2543fd5c3,46ae69081fd32d16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:17:06.624170+08:00, lastProcessTime=2025-04-08T21:16:56.622527+08:00 2025-04-08 21:17:06.653 INFO 26087 --- [ocp-schedules-24,7c17ca2c921c85ea,89bbee89cb216c5d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:17:06.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:17:06.624170+08:00 2025-04-08 21:17:07.881 INFO 26087 --- [ocp-schedules-4,81958ccdb34a5aa5,f27d1f856e4ee20f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:17:07.882 INFO 26087 --- [ocp-schedules-4,81958ccdb34a5aa5,f27d1f856e4ee20f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:17:08.317 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:08.318 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:08.606 WARN 26087 --- [ocp-async-10,335d632648b18bb6,dcf8d1b9ad08d2b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:08.606 INFO 26087 --- [ocp-async-5,326b4ad6498d89fd,c80dc3cebd9b1240] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:09.355 INFO 26087 --- [prometheus-minute-schedule-1,4da3254de0ba140d,7c74cd9f0c5ebd23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:09.403 INFO 26087 --- [ocp-schedules-11,e11d1005616ca53f,056276df12ebec99] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:17:10.001 INFO 26087 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:17:10.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:17:10.601 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 7, ruleSize = 7. 2025-04-08 21:17:10.606 WARN 26087 --- [ocp-async-3,a3fdc172ceb591c9,1b2f789682a13649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:10.606 INFO 26087 --- [ocp-async-8,2dbff171f4b78907,bdd3d174e8db40b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:11.318 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:11.318 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:12.355 INFO 26087 --- [prometheus-minute-schedule-1,7396f33f77ea172d,868711c7ada930fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:12.417 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118160, nodeCount=376, affectRows=376 2025-04-08 21:17:12.419 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118160, nodeCount=373, validCount=373 2025-04-08 21:17:12.606 WARN 26087 --- [ocp-async-6,799a4571cce848d2,88e81d2303b85a85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:12.606 INFO 26087 --- [ocp-async-9,cac01fb978f4a48c,236a6fe2289e2c35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:14.318 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:14.319 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:14.606 WARN 26087 --- [ocp-async-10,b9d5b3970af528de,064508ed103830c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:14.606 INFO 26087 --- [ocp-async-2,f2b65d15da0e0b0a,e751560c9ab3725c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:15.355 INFO 26087 --- [prometheus-minute-schedule-1,7ea5db4998936ff6,9baa3c1d3628a44c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:16.517 INFO 26087 --- [ocp-schedules-17,ad50442947ee0cc6,8bae8d504d1c63b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:17:16.517 INFO 26087 --- [ocp-schedules-6,8e0a70fa9a956ddd,dde689df29076232] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:17:16.521 INFO 26087 --- [ocp-schedules-19,28911c2c96d5fa5a,33f62dcc4fe6e699] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:17:16.521 INFO 26087 --- [ocp-schedules-19,28911c2c96d5fa5a,33f62dcc4fe6e699] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:17:16.521 INFO 26087 --- [ocp-schedules-8,dba1589ac0ff30b4,a15d55480513f425] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:17:16.523 INFO 26087 --- [ocp-schedules-15,07a277be60f29d07,e1bbec15fabade03] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:17:16.523 INFO 26087 --- [ocp-schedules-7,a6c0ad16cd99e1f7,1c9db23d8ca79eaa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:17:16.523 INFO 26087 --- [ocp-schedules-15,07a277be60f29d07,e1bbec15fabade03] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:17:16.523 INFO 26087 --- [ocp-schedules-7,a6c0ad16cd99e1f7,1c9db23d8ca79eaa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:17:16.524 INFO 26087 --- [ocp-schedules-6,8e0a70fa9a956ddd,dde689df29076232] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:17:16.524 INFO 26087 --- [ocp-schedules-11,ec69d23da87dcd6d,9245e11cd5b136ac] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:17:16.524 INFO 26087 --- [ocp-schedules-9,ab405bae4ee421e7,3755ce2ed561719f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:17:16.527 INFO 26087 --- [ocp-schedules-17,ad50442947ee0cc6,8bae8d504d1c63b1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:17:16.529 INFO 26087 --- [ocp-schedules-4,b5742120289ca8d2,17901a9469c07ff9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:17:16.551 INFO 26087 --- [ocp-async-3,3b0ecd5d6befff09,315d373d8b301641] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:17:16.582 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:17:16.582 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:17:16.585 INFO 26087 --- [ocp-schedules-20,8e54d6447c29f617,8cc32622dc9de031] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1498,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":374,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":375} 2025-04-08 21:17:16.585 INFO 26087 --- [ocp-schedules-20,8e54d6447c29f617,8cc32622dc9de031] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:17:16.587 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:17:16.587 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:17:16.587 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:17:16.588 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:17:16.588 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:17:16.588 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:17:16.588 INFO 26087 --- [ocp-schedules-26,a1093b6aef63fce0,0c31b793dbb241bd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:17:16.590 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:17:16.590 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:17:16.590 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:17:16.592 INFO 26087 --- [ocp-schedules-18,c123564d20e5b3e1,21f8ba1714360db2] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:17:16.605 INFO 26087 --- [ocp-schedules-29,07fe2b340cae9a0d,b2148677364ed128] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:17:16.607 WARN 26087 --- [ocp-async-6,8d1eb6b9acbff70d,4eaea99da2c5585d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:16.607 INFO 26087 --- [ocp-async-1,605f3d9863676fa4,90c90702a0df31f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:16.617 INFO 26087 --- [ocp-schedules-15,37e84f6283eaf010,8744f10605fe8f97] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:17:16.617 INFO 26087 --- [ocp-schedules-15,37e84f6283eaf010,8744f10605fe8f97] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:17:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:17:16.623673+08:00, lastProcessTime=2025-04-08T21:17:06.624170+08:00 2025-04-08 21:17:16.634 INFO 26087 --- [ocp-schedules-16,20f8953d8434fdbc,06b519a7f7fc9510] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:17:16.635 INFO 26087 --- [ocp-schedules-6,318291a38ddada10,dc928160b3c3a0d3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:17:16.638 INFO 26087 --- [ocp-schedules-11,dcc48522e342c102,e5e90259e60c8961] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:17:16.638 INFO 26087 --- [ocp-schedules-4,302286f222d7c202,f31898d36b00541f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:17:16.638 INFO 26087 --- [ocp-schedules-4,302286f222d7c202,f31898d36b00541f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:17:16.639 INFO 26087 --- [ocp-schedules-9,36c47380a4201e0b,badfe63ec7ff4480] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:17:16.639+08:00, clusters=[] 2025-04-08 21:17:16.645 INFO 26087 --- [ocp-schedules-17,abdb32351e3e2c8f,c41250e2f1f4d653] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:17:16.645+08:00, clusters=[] 2025-04-08 21:17:16.654 INFO 26087 --- [ocp-schedules-5,89db5029de14a441,bcec49ae5c56dbe4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:17:16.663 INFO 26087 --- [ocp-schedules-23,abf13f39f17c4a46,7d1f41922ba0bed2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:17:16.663 INFO 26087 --- [ocp-schedules-23,abf13f39f17c4a46,7d1f41922ba0bed2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:17:16.665 INFO 26087 --- [ocp-schedules-23,abf13f39f17c4a46,7d1f41922ba0bed2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:17:16.665 INFO 26087 --- [ocp-schedules-23,abf13f39f17c4a46,7d1f41922ba0bed2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:17:16.671 INFO 26087 --- [ocp-schedules-12,a896893254ef176e,c0eb0faaf5557265] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:17:16.672 INFO 26087 --- [ocp-schedules-12,a896893254ef176e,c0eb0faaf5557265] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:17:16.672 INFO 26087 --- [ocp-schedules-12,a896893254ef176e,c0eb0faaf5557265] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:17:16.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:17:16.623673+08:00 2025-04-08 21:17:16.753 INFO 26087 --- [ocp-schedules-22,28e79f50d0d91304,acd612e33e6e3833] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:17:16.824 INFO 26087 --- [ocp-schedules-25,4e20f9fb5dc93838,3542c21a711118b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:17:16.829 INFO 26087 --- [ocp-schedules-21,de41a88fa07b981d,615d2a03ecdf2bf0] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:17:16.835 INFO 26087 --- [ocp-schedules-13,7c7dd5cdfed4f1b6,af28b37aab8cdb5c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:17:16.835 INFO 26087 --- [ocp-schedules-13,7c7dd5cdfed4f1b6,af28b37aab8cdb5c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:17:16.835 INFO 26087 --- [ocp-schedules-13,7c7dd5cdfed4f1b6,af28b37aab8cdb5c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:17:16.835 INFO 26087 --- [ocp-schedules-13,7c7dd5cdfed4f1b6,af28b37aab8cdb5c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:17:16.835 INFO 26087 --- [ocp-schedules-13,7c7dd5cdfed4f1b6,af28b37aab8cdb5c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:17:16.835 INFO 26087 --- [ocp-schedules-13,7c7dd5cdfed4f1b6,af28b37aab8cdb5c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:17:16.837 INFO 26087 --- [ocp-schedules-28,834f49a2277a2a83,8cc7268c2dc70fe3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:17:17.188 INFO 26087 --- [ocp-schedules-30,3d1e1d9b03b85830,1384f9e35fd26a9a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:17:17.200 INFO 26087 --- [ocp-schedules-30,3d1e1d9b03b85830,1384f9e35fd26a9a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:17:17.200 INFO 26087 --- [ocp-schedules-30,3d1e1d9b03b85830,1384f9e35fd26a9a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:17:17.235 INFO 26087 --- [ocp-schedules-29,32cbb186e1ba8ed2,2e6ffea2827039f1] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,C_sgkydb-1744016068-PRIMARY,C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,C_sgkydb-1744016068/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65, current targets ,,,,,,,C_sgkydb-1744016068-PRIMARY,C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,C_sgkydb-1744016068/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.319 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:17.320 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:17.545 INFO 26087 --- [ocp-schedules-16,005ee61b14f1b364,370c74f05a1ccef9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:17:17.545 INFO 26087 --- [ocp-schedules-16,005ee61b14f1b364,370c74f05a1ccef9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:17:17.576 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:17:17.577 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:17:17.578 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:17:17.579 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:17:17.580 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:17:17.581 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:17:17.582 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:17:17.583 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:17:17.584 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.585 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:17:17.586 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:17:17.587 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:17:17.588 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:17:17.589 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:17:17.590 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:17:17.591 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:17:17.592 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:17:17.592 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_sgkydb-1744016068-PRIMARY 2025-04-08 21:17:17.592 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:17:17.592 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:17:17.592 INFO 26087 --- [ocp-schedules-27,43add5b5802174e4,d6faa8f33e6cf767] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:17:17.899 INFO 26087 --- [ocp-schedules-17,a0384b5deb03f020,6a4efa895df8c975] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:17:17.900 INFO 26087 --- [ocp-schedules-17,a0384b5deb03f020,6a4efa895df8c975] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:17:18.355 INFO 26087 --- [prometheus-minute-schedule-1,17bceaf06002fd09,b5c6352c8c2168b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:18.607 WARN 26087 --- [ocp-async-10,c8d9aeec269a7c9b,45f60134cf4abbe7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:18.607 INFO 26087 --- [ocp-async-1,d032ace8aca92ed1,d6b17814f490d95d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:19.421 INFO 26087 --- [ocp-schedules-8,f9ee0a62babeba4c,77948c4b68fafc8b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:17:20.320 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:20.320 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:17:20.607 WARN 26087 --- [ocp-async-5,16e5b75e271474a7,99e959a7e3624c2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:20.607 INFO 26087 --- [ocp-async-1,3bafb58e5b560bbb,cd8f0a08dfeda5be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:21.355 INFO 26087 --- [prometheus-minute-schedule-1,126f9d7002f7da7e,a1c65733d564e6ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:22.607 WARN 26087 --- [ocp-async-6,1d32cc06168dfe5a,9f5eea8e1aa7cfd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:22.607 INFO 26087 --- [ocp-async-1,574ab8aeb2d42880,7667e2731196ede0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:23.321 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:23.321 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:24.355 INFO 26087 --- [prometheus-minute-schedule-1,e7df08c0f4713305,507dfa3d645e1a9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:24.607 WARN 26087 --- [ocp-async-4,00cd60fceb30265c,85d8089b53d540d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:24.607 INFO 26087 --- [ocp-async-10,5e588d15ac0a8537,932ae4c623a35a66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:26.321 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:26.322 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:26.551 INFO 26087 --- [ocp-async-5,2eaa51e0fcf0a344,3a43568412fd6968] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:17:26.607 WARN 26087 --- [ocp-async-7,a631177cc28d5dc3,fec46f0dc4f87647] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:26.607 INFO 26087 --- [ocp-async-2,a03c1ff53caaa5b6,890af6211350aff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:17:26.623115+08:00, lastProcessTime=2025-04-08T21:17:16.623673+08:00 2025-04-08 21:17:26.655 INFO 26087 --- [ocp-schedules-28,f6a7ebd97be34c99,b1e9da54551fa849] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:17:26.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:17:26.623115+08:00 2025-04-08 21:17:27.355 INFO 26087 --- [prometheus-minute-schedule-1,c709aa5fd2412593,d5dfb78020c3bbf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:27.908 INFO 26087 --- [ocp-schedules-13,633e94feff891500,752b4e8049f287a7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:17:27.910 INFO 26087 --- [ocp-schedules-13,633e94feff891500,752b4e8049f287a7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:17:28.607 INFO 26087 --- [ocp-async-2,aedf5a700e8339ad,91cea87073e309d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:28.608 WARN 26087 --- [ocp-async-4,8052fa43c5950914,cc706b926fe3afac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:29.322 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:29.322 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:29.489 INFO 26087 --- [ocp-schedules-24,ab2236b6f60b94ee,61dbbb9292bda86a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:17:30.355 INFO 26087 --- [prometheus-minute-schedule-1,6c567fd6a1faa3ee,a7486b5c184fffd4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:30.608 WARN 26087 --- [ocp-async-2,bc58942bfff27d47,765ece32ad9eb0af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:30.608 INFO 26087 --- [ocp-async-10,d52e6e4338dadb57,a22e52d3e59032c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:32.322 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:32.323 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:32.608 WARN 26087 --- [ocp-async-6,583e05c027bcafc7,56294627557ba5c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:32.608 INFO 26087 --- [ocp-async-10,d03bb08a9b2056a3,4ac2abdf8fef9e32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:33.355 INFO 26087 --- [prometheus-minute-schedule-1,f5b949b132456cf0,cb4102d7322f1810] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:34.608 WARN 26087 --- [ocp-async-8,472ed644745cea00,155a628d67a5b144] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:34.608 INFO 26087 --- [ocp-async-4,b83d3f9db7e26810,8d14d0bf0655c00d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:34.978 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:17:34.980 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=2ms, period=120000ms 2025-04-08 21:17:35.323 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:35.324 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:36.355 INFO 26087 --- [prometheus-minute-schedule-1,dd7ff547f4be7008,390ca79f2bc0c13a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:36.551 INFO 26087 --- [ocp-async-4,1cc8d8f36b65ff46,cee9f5f5eb10d393] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:17:36.608 WARN 26087 --- [ocp-async-5,2f7d45d5a87b1ca9,91bbd1cf33fd1c10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:36.608 INFO 26087 --- [ocp-async-9,f663d1c109d0f16a,dccdcee8ebddcff7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:17:36.622924+08:00, lastProcessTime=2025-04-08T21:17:26.623115+08:00 2025-04-08 21:17:36.656 INFO 26087 --- [ocp-schedules-17,d152cdcfc0f5ea3b,2be5e87c983ba2c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:17:36.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:17:36.622924+08:00 2025-04-08 21:17:37.048 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,6f94484cf4e676d8,c84445f8ece9d0be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.11.12.102, traceId=6f94484cf4e676d8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,OBCLUSTER,] 2025-04-08 21:17:37.054 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,6f94484cf4e676d8,c84445f8ece9d0be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=6f94484cf4e676d8, duration=10 ms] 2025-04-08 21:17:37.076 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,244968393a838733,936f6d3296e37dba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.11.12.102, traceId=244968393a838733, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:17:37.111 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,244968393a838733,936f6d3296e37dba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.11.12.102, traceId=244968393a838733, duration=41 ms] 2025-04-08 21:17:37.201 INFO 26087 --- [ocp-schedules-20,288bc1937f7a1ab5,66e415459c008ca2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:17:37.205 INFO 26087 --- [ocp-schedules-20,288bc1937f7a1ab5,66e415459c008ca2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:17:37.205 INFO 26087 --- [ocp-schedules-20,288bc1937f7a1ab5,66e415459c008ca2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:17:37.467 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,12a9d0164eec64e1,e37d6f9c3d66c49c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A17%3A37%2B08%3A00&endTime=2025-04-08T21%3A17%3A37%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=12a9d0164eec64e1, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:17:37+08:00,2025-04-08T21:17:37+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:17:37.468 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-1,12a9d0164eec64e1,e37d6f9c3d66c49c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:17:37.468 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-1,12a9d0164eec64e1,e37d6f9c3d66c49c] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114665 2025-04-08 21:17:37.469 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,12a9d0164eec64e1,e37d6f9c3d66c49c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=12a9d0164eec64e1, duration=5 ms] 2025-04-08 21:17:37.475 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,42292f9dc6428371,cc801d56cd1a7201] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A17%3A37%2B08%3A00&endTime=2025-04-08T21%3A17%3A37%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=42292f9dc6428371, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:17:37+08:00,2025-04-08T21:17:37+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:17:37.476 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,42292f9dc6428371,cc801d56cd1a7201] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_rt 2025-04-08 21:17:37.476 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,42292f9dc6428371,cc801d56cd1a7201] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_rt, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114665 2025-04-08 21:17:37.478 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,42292f9dc6428371,cc801d56cd1a7201] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=42292f9dc6428371, duration=6 ms] 2025-04-08 21:17:37.496 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c3b09c5899fbb02,125c9fee202e26cd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A17%3A37%2B08%3A00&endTime=2025-04-08T21%3A17%3A37%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=8c3b09c5899fbb02, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:17:37+08:00,2025-04-08T21:17:37+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:17:37.496 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c3b09c5899fbb02,125c9fee202e26cd] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = tps_rt 2025-04-08 21:17:37.496 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c3b09c5899fbb02,125c9fee202e26cd] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = tps_rt, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114665 2025-04-08 21:17:37.497 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c3b09c5899fbb02,125c9fee202e26cd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=8c3b09c5899fbb02, duration=5 ms] 2025-04-08 21:17:37.530 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,cf8f5d19340da317,5481e37f5d12ca08] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A17%3A37%2B08%3A00&endTime=2025-04-08T21%3A17%3A37%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=cf8f5d19340da317, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:17:37+08:00,2025-04-08T21:17:37+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:17:37.530 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,cf8f5d19340da317,5481e37f5d12ca08] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = tps 2025-04-08 21:17:37.531 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,cf8f5d19340da317,5481e37f5d12ca08] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = tps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114665 2025-04-08 21:17:37.532 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,cf8f5d19340da317,5481e37f5d12ca08] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=cf8f5d19340da317, duration=6 ms] 2025-04-08 21:17:37.915 INFO 26087 --- [ocp-schedules-26,03947bb33df74f81,0ade6f38bc5fd50e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:17:37.916 INFO 26087 --- [ocp-schedules-26,03947bb33df74f81,0ade6f38bc5fd50e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:17:37.965 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,1c65711a9a7e4024,05ff079922b47e99] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=1c65711a9a7e4024, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:17:37.968 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,1c65711a9a7e4024,05ff079922b47e99] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=1c65711a9a7e4024, duration=6 ms] 2025-04-08 21:17:38.288 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,cbb8394378a159db,5a2e3fc1a09c50c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=cbb8394378a159db, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:17:38.288 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,3c5a67175d5d32e6,6b2b51fd7c97db1f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=3c5a67175d5d32e6, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:17:38.288 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,a520219717752e28,c78916c349ab7eda] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A17%3A38%2B08%3A00&endTime=2025-04-08T21%3A17%3A38%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=a520219717752e28, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:17:38+08:00,2025-04-08T21:17:38+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:17:38.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,3c5a67175d5d32e6,6b2b51fd7c97db1f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=3c5a67175d5d32e6, duration=4 ms] 2025-04-08 21:17:38.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,62b6ac947b2d0a75,e611046eb528b57b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=62b6ac947b2d0a75, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:17:38.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,62bbd33c030d171b,b7aa124ad4fb1d3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=62bbd33c030d171b, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:17:38.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,62bbd33c030d171b,b7aa124ad4fb1d3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=62bbd33c030d171b, duration=4 ms] 2025-04-08 21:17:38.290 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,cbb8394378a159db,5a2e3fc1a09c50c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=cbb8394378a159db, duration=5 ms] 2025-04-08 21:17:38.290 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-4,a520219717752e28,c78916c349ab7eda] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:17:38.290 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-4,a520219717752e28,c78916c349ab7eda] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744114665 2025-04-08 21:17:38.291 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,a520219717752e28,c78916c349ab7eda] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=a520219717752e28, duration=6 ms] 2025-04-08 21:17:38.293 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,62b6ac947b2d0a75,e611046eb528b57b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=62b6ac947b2d0a75, duration=8 ms] 2025-04-08 21:17:38.316 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,ffed64ec25644bbd,9cf92dcb3ea4fe7f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=ffed64ec25644bbd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:17:38.322 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,ffed64ec25644bbd,9cf92dcb3ea4fe7f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=ffed64ec25644bbd, duration=16 ms] 2025-04-08 21:17:38.324 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:38.324 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:38.608 WARN 26087 --- [ocp-async-7,5869bdf5b98ce0d6,d71bacca37b08814] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:38.608 INFO 26087 --- [ocp-async-10,db13abc42aa8d037,f07b72038ec2cbe1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:39.260 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,c75065570cbf92ae,70105eab2d133764] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=c75065570cbf92ae, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:17:39.263 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,c75065570cbf92ae,70105eab2d133764] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=c75065570cbf92ae, duration=6 ms] 2025-04-08 21:17:39.355 INFO 26087 --- [prometheus-minute-schedule-1,e639e8445ad3fd67,fa86d1ccb07f0aa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:39.506 INFO 26087 --- [ocp-schedules-15,09d0e2c5c3a7eeac,311c7a5857f2c999] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:17:40.001 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:17:40.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:17:40.608 WARN 26087 --- [ocp-async-3,ae9fdf8748eaf1fc,75af8498b5633186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:40.608 INFO 26087 --- [ocp-async-1,1844d45cbac16e46,fc7ecfdf251d4fff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:41.324 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:41.325 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:42.355 INFO 26087 --- [prometheus-minute-schedule-1,44ef3a7cce8c976e,c3dbb4c4d4d46761] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:42.608 WARN 26087 --- [ocp-async-5,bf5f5d29635ec5e5,1a512abc5c7fbd63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:42.609 INFO 26087 --- [ocp-async-4,733ab12dabbb0bbd,e493e77d8b077a64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:44.325 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:44.326 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:44.609 WARN 26087 --- [ocp-async-4,a444a99457a56d64,a876267543953c0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:44.609 INFO 26087 --- [ocp-async-8,9e3d619e3b8a8d12,f67fe3862efe108c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:45.355 INFO 26087 --- [prometheus-minute-schedule-1,5fc09ec4dcf9c1bb,32e05ade01d5a7cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:46.521 INFO 26087 --- [ocp-schedules-17,79a28cb6cd683eae,7901a8728649e9c0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:17:46.521 INFO 26087 --- [ocp-schedules-20,dd3e9489db7cdf2f,d3da441b82a777a4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:17:46.521 INFO 26087 --- [ocp-schedules-20,dd3e9489db7cdf2f,d3da441b82a777a4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:17:46.523 INFO 26087 --- [ocp-schedules-2,6b9ff6dfad9849b9,c35f54b631e399c7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:17:46.523 INFO 26087 --- [ocp-schedules-21,e4d8a590f6f643ac,7dc20116bf3c224a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:17:46.523 INFO 26087 --- [ocp-schedules-21,e4d8a590f6f643ac,7dc20116bf3c224a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:17:46.523 INFO 26087 --- [ocp-schedules-18,2e749a3a24178799,9a2e70bccdf9dd10] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:17:46.523 INFO 26087 --- [ocp-schedules-18,2e749a3a24178799,9a2e70bccdf9dd10] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:17:46.524 INFO 26087 --- [ocp-schedules-4,1c63668cecec229e,a9e684f89763904d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:17:46.524 INFO 26087 --- [ocp-schedules-22,9d2e51f1614d518c,5d9ade4a5df0e2a9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:17:46.524 INFO 26087 --- [ocp-schedules-3,e9e29a166c8cf656,b216c4a26c2d0905] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:17:46.551 INFO 26087 --- [ocp-async-6,6614067fd4b7f83f,ccf9edaddbcff563] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:17:46.606 INFO 26087 --- [ocp-schedules-11,60bd32c90a5737cf,b18a37b4d6058071] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:17:46.609 WARN 26087 --- [ocp-async-10,68418e5ac8f6856e,93b5320da630ea38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:46.609 INFO 26087 --- [ocp-async-1,aa0dffad91065d92,e66b8281b7cda825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:17:46.623447+08:00, lastProcessTime=2025-04-08T21:17:36.622924+08:00 2025-04-08 21:17:46.657 INFO 26087 --- [ocp-schedules-7,a9437a655544b3d4,4b17c92950321ad9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:17:46.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:17:46.623447+08:00 2025-04-08 21:17:47.326 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:47.327 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:47.958 INFO 26087 --- [ocp-schedules-15,3e6ef84c5d150172,a674963f5247947d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:17:47.959 INFO 26087 --- [ocp-schedules-15,3e6ef84c5d150172,a674963f5247947d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:17:48.355 INFO 26087 --- [prometheus-minute-schedule-1,b5934b6a5a2f9ec7,ce5c3aaa81e4de36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:48.609 WARN 26087 --- [ocp-async-5,8fb35a2bf18dfb0d,a531547d11cc128a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:48.609 INFO 26087 --- [ocp-async-9,f62470e80f063848,2ebf1e0cb812ba43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:49.527 INFO 26087 --- [ocp-schedules-15,602ca34fdae14b2b,8755917fe20ae5d5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:17:50.327 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:50.327 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:50.609 INFO 26087 --- [ocp-async-4,ab16740d843b52ff,630423ab7a54216f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:50.609 WARN 26087 --- [ocp-async-8,ed1973e2a3204a5b,38154829a3e6d24a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:51.355 INFO 26087 --- [prometheus-minute-schedule-1,c4d535181c078c15,71d13413c4dc2667] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:52.609 INFO 26087 --- [ocp-async-6,e58ee860165a45ad,c99158378afddc33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:52.609 WARN 26087 --- [ocp-async-3,676df8913c6b192e,641aecd0d8ad965c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:53.327 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:53.328 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:54.355 INFO 26087 --- [prometheus-minute-schedule-1,92ba473eaff7ea7b,09b6dc17f56be4a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:54.609 WARN 26087 --- [ocp-async-5,4d531407df8f6463,a93c6523976ea8de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:54.609 INFO 26087 --- [ocp-async-2,ae9aadf2477f5cd8,6878d7921d8d6333] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:56.328 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:56.329 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:56.551 INFO 26087 --- [ocp-async-2,18315d520013b192,d4236cf0a447ea0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:17:56.567 INFO 26087 --- [ocp-schedules-28,a23ac53c780efe23,d7d05b960a7aafce] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:17:56.610 WARN 26087 --- [ocp-async-4,837e3462c0af1bc8,cc3dcb78b22846eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:56.610 INFO 26087 --- [ocp-async-7,25643cd6525c4a5d,9a1dbfe3bd7981aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:17:56.623543+08:00, lastProcessTime=2025-04-08T21:17:46.623447+08:00 2025-04-08 21:17:56.658 INFO 26087 --- [ocp-schedules-23,50e7f48d4df3a265,d295506c40deffe6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:17:56.751 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:17:56.623543+08:00 2025-04-08 21:17:57.206 INFO 26087 --- [ocp-schedules-14,4b91a604e951b81d,4e1607c399887c20] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:17:57.210 INFO 26087 --- [ocp-schedules-14,4b91a604e951b81d,4e1607c399887c20] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:17:57.210 INFO 26087 --- [ocp-schedules-14,4b91a604e951b81d,4e1607c399887c20] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:17:57.355 INFO 26087 --- [prometheus-minute-schedule-1,6a63ad3fc1b69368,ded103ed8b56abcd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118280, interval=60 2025-04-08 21:17:57.963 INFO 26087 --- [ocp-schedules-16,dfe7feec1708a338,6d373796281a62a0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:17:57.964 INFO 26087 --- [ocp-schedules-16,dfe7feec1708a338,6d373796281a62a0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:17:58.610 WARN 26087 --- [ocp-async-10,eee69b7b48c4b48f,3d0223060b967c5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:17:58.610 INFO 26087 --- [ocp-async-7,00ae6d02b49587f5,a7f1c0bedb75ab69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:17:59.329 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:17:59.329 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:17:59.544 INFO 26087 --- [ocp-schedules-8,d31534aa75a92bd9,4aaad7e3739a18a7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:18:00.355 INFO 26087 --- [prometheus-minute-schedule-1,5d12c2b2531392a4,27c1f33d37410e18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:00.610 WARN 26087 --- [ocp-async-5,1855c89de8ba8d38,6dd6bfdd0c03e622] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:00.610 INFO 26087 --- [ocp-async-1,a1b29349b98edd47,46c40d9ffc228966] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:02.330 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:02.330 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:02.610 WARN 26087 --- [ocp-async-8,5026520ec6bd0b42,6808227b9bc6df4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:02.610 INFO 26087 --- [ocp-async-2,544524359210c363,c3e71677259b74d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:03.355 INFO 26087 --- [prometheus-minute-schedule-1,43cd94450dc8e886,c35ac3f298d62bce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:04.610 WARN 26087 --- [ocp-async-6,9d42390b993b901b,1b7569a590ef74c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:04.610 INFO 26087 --- [ocp-async-3,223ebc369e3f1440,5d011c963c4a7859] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:05.330 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:05.331 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:06.355 INFO 26087 --- [prometheus-minute-schedule-1,076d6ef96e4f14a6,f4adbf11e69d91e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:06.355 INFO 26087 --- [prometheus-cache-monitor1,da2521786050152d,9befab5f567d6607] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1498,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":375} 2025-04-08 21:18:06.355 INFO 26087 --- [prometheus-cache-monitor1,da2521786050152d,9befab5f567d6607] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:18:06.418 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:14:06.378, end=2025-04-08T21:16:06.406} 2025-04-08 21:18:06.420 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:18:06.420 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:18:06.551 INFO 26087 --- [ocp-async-7,72a732018e576703,59d6af419328f3fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:18:06.610 INFO 26087 --- [ocp-async-9,3655c5a15157a6ab,ac31562c9362304e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:06.610 WARN 26087 --- [ocp-async-1,fb7da926dfb5871a,7246df2b6732aa90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:18:06.623744+08:00, lastProcessTime=2025-04-08T21:17:56.623543+08:00 2025-04-08 21:18:06.660 INFO 26087 --- [ocp-schedules-8,9a309593f3a03f67,439f9b0eaf774ce7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:18:06.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:18:06.623744+08:00 2025-04-08 21:18:07.967 INFO 26087 --- [ocp-schedules-23,a2f3ea2ea10745f8,8d950e3505d16a28] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:18:07.968 INFO 26087 --- [ocp-schedules-23,a2f3ea2ea10745f8,8d950e3505d16a28] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:18:08.331 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:08.332 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:08.610 WARN 26087 --- [ocp-async-4,cd7e03c3055257ee,3547fa517184c247] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:08.610 INFO 26087 --- [ocp-async-2,a4fe8c73ce65a53c,2ae1d05d3bf787b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:09.355 INFO 26087 --- [prometheus-minute-schedule-1,4ed0ab0272b58f4c,caad84c8cceb8d97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:09.566 INFO 26087 --- [ocp-schedules-19,4035651d0bd4e777,9e23ffe40aedef38] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:18:10.001 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:18:10.001 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:18:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:18:10.610 WARN 26087 --- [ocp-async-10,33230b333f5760c7,31392f11289641c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:10.611 INFO 26087 --- [ocp-async-6,51a82489cdbba0e9,82750264f6ba9bcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:11.332 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:11.332 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:12.355 INFO 26087 --- [prometheus-minute-schedule-1,9b1c13c12731d24e,20131d29d7943f98] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:12.408 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118220, nodeCount=378, affectRows=378 2025-04-08 21:18:12.410 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118220, nodeCount=377, validCount=377 2025-04-08 21:18:12.611 WARN 26087 --- [ocp-async-5,4485a8b09e0b7558,2199474fbc643816] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:12.611 INFO 26087 --- [ocp-async-1,e4e3a8b02317ac39,e2f51412e5f07e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:14.332 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:14.333 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:14.611 WARN 26087 --- [ocp-async-1,bc32d7f029bf033e,f3ac693a57d911cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:14.611 INFO 26087 --- [ocp-async-8,7de641a1816abbcc,d66817f203a31683] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:15.355 INFO 26087 --- [prometheus-minute-schedule-1,93581015b21e2a6c,3ccc2970fffa26a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:16.518 INFO 26087 --- [ocp-schedules-29,afc86b90d918c66b,7147b249d1cea88e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:18:16.518 INFO 26087 --- [ocp-schedules-24,e73aa0c36cef2cc9,829b6f607c2cab71] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:18:16.521 INFO 26087 --- [ocp-schedules-12,67cbb488bfc3e2c3,7701888ec2180de6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:18:16.521 INFO 26087 --- [ocp-schedules-26,ed808be6f133e2e0,04ed67e9e9a84f41] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:18:16.521 INFO 26087 --- [ocp-schedules-26,ed808be6f133e2e0,04ed67e9e9a84f41] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:18:16.523 INFO 26087 --- [ocp-schedules-29,afc86b90d918c66b,7147b249d1cea88e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:18:16.523 INFO 26087 --- [ocp-schedules-23,caee63271c7e484e,4f05f233e953379f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:18:16.523 INFO 26087 --- [ocp-schedules-23,caee63271c7e484e,4f05f233e953379f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:18:16.523 INFO 26087 --- [ocp-schedules-14,a42ade87cca0ec7a,b2d342caae5b7198] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:18:16.523 INFO 26087 --- [ocp-schedules-14,a42ade87cca0ec7a,b2d342caae5b7198] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:18:16.524 INFO 26087 --- [ocp-schedules-26,4bc5a88966cc191b,29c4bb44ed1a8111] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:18:16.524 INFO 26087 --- [ocp-schedules-19,e8e14b17fc88a36d,9d196cc9a725e7d2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:18:16.524 INFO 26087 --- [ocp-schedules-16,77c146e446b83f98,ff0f5175af78795d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:18:16.525 INFO 26087 --- [ocp-schedules-24,e73aa0c36cef2cc9,829b6f607c2cab71] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:18:16.534 INFO 26087 --- [ocp-schedules-9,363af8fecae183e8,68d3dfe61b68530b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:18:16.535 INFO 26087 --- [ocp-schedules-6,6c32513dda140445,66134eda6f94f51e] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:18:16.535 INFO 26087 --- [ocp-schedules-6,6c32513dda140445,66134eda6f94f51e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:18:16.536 INFO 26087 --- [ocp-schedules-17,f0e05fd1137f051d,efc97f50e131caa8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:18:16.551 INFO 26087 --- [ocp-async-10,01a4ce9ed3ebc482,2c0f00d2435b0acb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:18:16.553 INFO 26087 --- [pool-overview-container-7,e1a317db11460a6c,9a2df27be8360076] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:18:16.553 INFO 26087 --- [pool-overview-container-6,3ae01c51791cbc18,96c6025f6b77d690] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:18:16.590 INFO 26087 --- [ocp-schedules-2,289dbc926c11cb5b,921b29b72214b63d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:18:16.608 INFO 26087 --- [ocp-schedules-20,337e4e7de5d7d287,d64ba127e892f2d4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:18:16.611 WARN 26087 --- [ocp-async-6,9859aa9a0d785229,8e5639a2bb909aee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:16.611 INFO 26087 --- [ocp-async-3,6020554123ea1b7c,f4b1d3bf6ad30d79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:16.620 INFO 26087 --- [ocp-schedules-23,3efe92a0d2892fe1,ad2dafe02a68c655] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:18:16.620 INFO 26087 --- [ocp-schedules-23,3efe92a0d2892fe1,ad2dafe02a68c655] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:18:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:18:16.622411+08:00, lastProcessTime=2025-04-08T21:18:06.623744+08:00 2025-04-08 21:18:16.640 INFO 26087 --- [ocp-schedules-24,cb8f440bd6ad7c79,5e63ae6911b8b25e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:18:16.640 INFO 26087 --- [ocp-schedules-24,cb8f440bd6ad7c79,5e63ae6911b8b25e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:18:16.640 INFO 26087 --- [ocp-schedules-16,9698542de8d7a8ae,69d139758183c1db] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:18:16.640+08:00, clusters=[] 2025-04-08 21:18:16.641 INFO 26087 --- [ocp-schedules-19,6e0a9d7900f2d5cc,58b6d0f0b05ae968] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:18:16.647 INFO 26087 --- [ocp-schedules-6,13cc069befac863d,31c2c157a43e177a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:18:16.647+08:00, clusters=[] 2025-04-08 21:18:16.661 INFO 26087 --- [ocp-schedules-29,b474c3dfdaf4888d,6ea6f37b61f82179] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:18:16.666 INFO 26087 --- [ocp-schedules-17,51dbd5c587dba866,ae022bd4c43deff8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:18:16.666 INFO 26087 --- [ocp-schedules-17,51dbd5c587dba866,ae022bd4c43deff8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:18:16.667 INFO 26087 --- [ocp-schedules-17,51dbd5c587dba866,ae022bd4c43deff8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:18:16.667 INFO 26087 --- [ocp-schedules-17,51dbd5c587dba866,ae022bd4c43deff8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:18:16.673 INFO 26087 --- [ocp-schedules-22,40238835a7315ca2,78ca67995f8369b7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:18:16.674 INFO 26087 --- [ocp-schedules-22,40238835a7315ca2,78ca67995f8369b7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:18:16.674 INFO 26087 --- [ocp-schedules-22,40238835a7315ca2,78ca67995f8369b7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:18:16.721 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:18:16.622411+08:00 2025-04-08 21:18:16.760 INFO 26087 --- [ocp-schedules-21,d10b7b80bd1d8d85,8c9c1724ffd143c1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:18:16.822 INFO 26087 --- [ocp-schedules-5,b2fc46956213a4c9,ae6efc45361f8b47] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:18:16.822 INFO 26087 --- [ocp-schedules-5,b2fc46956213a4c9,ae6efc45361f8b47] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:18:16.823 INFO 26087 --- [ocp-schedules-5,b2fc46956213a4c9,ae6efc45361f8b47] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:18:16.824 INFO 26087 --- [ocp-schedules-25,5f1921a98129ad5d,673a639ce4818afe] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:18:16.837 INFO 26087 --- [ocp-schedules-10,b56355ce451c5224,6e036b5b21a46258] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:18:16.837 INFO 26087 --- [ocp-schedules-10,b56355ce451c5224,6e036b5b21a46258] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:18:16.837 INFO 26087 --- [ocp-schedules-10,b56355ce451c5224,6e036b5b21a46258] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:18:16.837 INFO 26087 --- [ocp-schedules-10,b56355ce451c5224,6e036b5b21a46258] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:18:16.837 INFO 26087 --- [ocp-schedules-10,b56355ce451c5224,6e036b5b21a46258] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:18:16.837 INFO 26087 --- [ocp-schedules-10,b56355ce451c5224,6e036b5b21a46258] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:18:16.839 INFO 26087 --- [ocp-schedules-15,efdfab9868407f31,eba2cdc654b77916] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:18:17.154 INFO 26087 --- [ocp-schedules-4,6310f664053f16eb,109f1da10bd3832c] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:18:17.211 INFO 26087 --- [ocp-schedules-2,2f5f5be3ba86064a,41f8643502e20469] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:18:17.214 INFO 26087 --- [ocp-schedules-2,2f5f5be3ba86064a,41f8643502e20469] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:18:17.214 INFO 26087 --- [ocp-schedules-2,2f5f5be3ba86064a,41f8643502e20469] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:18:17.333 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:17.334 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:17.548 INFO 26087 --- [ocp-schedules-16,ab56a5d17311bc65,77e30ad89b117020] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:18:17.548 INFO 26087 --- [ocp-schedules-16,ab56a5d17311bc65,77e30ad89b117020] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:18:17.972 INFO 26087 --- [ocp-schedules-13,743ead99ec3d56dc,e3fcedb02d7c1771] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:18:17.973 INFO 26087 --- [ocp-schedules-13,743ead99ec3d56dc,e3fcedb02d7c1771] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:18:18.355 INFO 26087 --- [prometheus-minute-schedule-1,3d1ba1f2627ddfbb,bd23bbb1e1eb48c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:18.611 WARN 26087 --- [ocp-async-3,a86ffd16bb9933c4,e0c8a82e2a587093] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:18.611 INFO 26087 --- [ocp-async-1,461a9ffbde583724,32e46be48d680957] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:19.583 INFO 26087 --- [ocp-schedules-3,b7a376dff9c47afe,4fd39313922c4ddc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:18:20.334 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:20.334 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:18:20.611 WARN 26087 --- [ocp-async-2,975eb1a08771b3de,43d29af4504925c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:20.611 INFO 26087 --- [ocp-async-8,4be6e06a034a71e4,1b10610a25db23ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:21.355 INFO 26087 --- [prometheus-minute-schedule-1,eced4db71e75b88a,bc12ec2e766d60b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:22.611 WARN 26087 --- [ocp-async-8,539e240ff63607f0,183cfa95174a2fec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:22.611 INFO 26087 --- [ocp-async-6,f10f29864104f4b5,59c0490efa1e6cef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:22.886 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:18:22.886 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:18:22.889 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:18:22.889 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:18:22.889 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:18:22.889 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:18:22.893 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:18:22.893 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:18:22.893 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:18:22.893 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:18:22.915 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:18:22.920 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:18:22.920 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:18:22.920 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:18:22.920 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:18:22.920 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:18:22.925 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:18:23.236 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:18:23.334 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:23.335 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:23.577 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.684144842S 2025-04-08 21:18:23.577 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:18:23.636 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:18:23.636 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:18:23.661 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:18:23.702 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:18:23.702 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:18:23.702 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:18:23.702 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:18:23.702 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:18:23.702 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:18:23.709 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:18:23.806 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] o.s.boot.SpringApplication : Started application in 0.917 seconds (JVM running for 1009.179) 2025-04-08 21:18:23.806 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.917274252S 2025-04-08 21:18:24.082 INFO 26087 --- [ocp-schedules-5,fef556c77228c99e,530fa901f881e68f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:18:24.355 INFO 26087 --- [prometheus-minute-schedule-1,3b4444262639514d,3aefa47ec60b7f63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:24.611 WARN 26087 --- [ocp-async-3,c8b0e16a49515f1c,af149f43d5ccf487] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:24.612 INFO 26087 --- [ocp-async-4,c19d0434947690b1,1c6a0511a0e95d05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:26.335 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:26.336 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:26.552 INFO 26087 --- [ocp-async-2,45309d5c53eefcdd,2a18e7be0e90b039] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:18:26.612 WARN 26087 --- [ocp-async-10,3aa5beefe9681978,54acaeb3d85ac782] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:26.612 INFO 26087 --- [ocp-async-1,ce38934ff5094eb4,e2ec5845864b662a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:18:26.624625+08:00, lastProcessTime=2025-04-08T21:18:16.622411+08:00 2025-04-08 21:18:26.662 INFO 26087 --- [ocp-schedules-14,26bae0d44b710135,f9932c733c3ae5bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:18:26.747 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:18:26.624625+08:00 2025-04-08 21:18:27.355 INFO 26087 --- [prometheus-minute-schedule-1,62f2e0eaa9caee25,dc4dee62bc292057] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:27.977 INFO 26087 --- [ocp-schedules-5,f8caceee22bcef5b,532158591a24863d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:18:27.978 INFO 26087 --- [ocp-schedules-5,f8caceee22bcef5b,532158591a24863d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:18:28.612 WARN 26087 --- [ocp-async-1,36a717811d45f11a,821617ab2e3f311f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:28.612 INFO 26087 --- [ocp-async-2,4ae9f29b1aca5b72,01e5a108543ff413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:29.336 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:29.336 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:29.616 INFO 26087 --- [ocp-schedules-22,aab35755acd01dc9,dd91c051687403e1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:18:30.355 INFO 26087 --- [prometheus-minute-schedule-1,55a66bf7efe03752,0c1003a2195898c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:30.612 WARN 26087 --- [ocp-async-4,fcaaae701beb3b5f,3b8ad0bde4a6e29c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:30.612 INFO 26087 --- [ocp-async-2,32b8ee353c803ea5,8c9fbbf618c0e216] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:32.337 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:32.337 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:32.612 WARN 26087 --- [ocp-async-5,2d7f849f3e734037,0dabcb9ad146b681] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:32.612 INFO 26087 --- [ocp-async-2,ec7e715e0f995557,a96070d30fae6135] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:33.355 INFO 26087 --- [prometheus-minute-schedule-1,fce8eb77d1b42791,370049b452abf309] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:34.612 WARN 26087 --- [ocp-async-1,5f2e23831440f24c,45185540150d3721] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:34.612 INFO 26087 --- [ocp-async-2,83994b4e83d2b4ef,c21d25e004cbf0b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:35.337 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:35.338 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:36.355 INFO 26087 --- [prometheus-minute-schedule-1,8df83158434a07b4,74f8603248e20fe8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:36.552 INFO 26087 --- [ocp-async-2,1c385c3754b5bce2,5c526febd7aea3c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:18:36.612 WARN 26087 --- [ocp-async-6,dd24f526e18339d0,84f5062a33b5a9d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:36.612 INFO 26087 --- [ocp-async-4,cd535caba4eedd70,ff84662c9db8edf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:18:36.623340+08:00, lastProcessTime=2025-04-08T21:18:26.624625+08:00 2025-04-08 21:18:36.663 INFO 26087 --- [ocp-schedules-23,385cb525f1729374,4a57c26cb6220052] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:18:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:18:36.623340+08:00 2025-04-08 21:18:37.216 INFO 26087 --- [ocp-schedules-30,06e1afe8d3a91ad1,0211c2899d778cd6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:18:37.235 INFO 26087 --- [ocp-schedules-30,06e1afe8d3a91ad1,0211c2899d778cd6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:18:37.235 INFO 26087 --- [ocp-schedules-30,06e1afe8d3a91ad1,0211c2899d778cd6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:18:37.985 INFO 26087 --- [ocp-schedules-19,81514174ac84dcf9,cb4282407aa47009] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:18:37.986 INFO 26087 --- [ocp-schedules-19,81514174ac84dcf9,cb4282407aa47009] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:18:38.338 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:38.338 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:38.613 WARN 26087 --- [ocp-async-4,e30e243d1e62e4c0,a2afd325c8862c4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:38.613 INFO 26087 --- [ocp-async-7,0fe90753a23cf170,951eec8a7a4445f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:39.355 INFO 26087 --- [prometheus-minute-schedule-1,a4225319e3998c15,4acd66bd2657936f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:39.633 INFO 26087 --- [ocp-schedules-28,51b278ab923270ee,8cd74d3fa94a5a5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:18:40.001 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:18:40.001 INFO 26087 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:18:40.613 WARN 26087 --- [ocp-async-3,e25a43800ef5d262,58f0756710398766] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:40.613 INFO 26087 --- [ocp-async-9,42458ce2fbdb5422,7a6762d4478d285c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:41.338 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:41.339 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:42.355 INFO 26087 --- [prometheus-minute-schedule-1,fae2942f3e2b18ec,cc41bc7306f4432a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:42.613 WARN 26087 --- [ocp-async-9,757a7a56170caf80,85c2a3a9c2738723] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:42.613 INFO 26087 --- [ocp-async-10,12d936456244832f,8d36bc15f079381e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:44.339 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:44.340 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:44.613 WARN 26087 --- [ocp-async-10,8e3b80d9cd3c26ef,6032c7d9896f5781] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:44.613 INFO 26087 --- [ocp-async-8,d5dd5b45eed59ff0,8cf54f65f198d2a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:45.355 INFO 26087 --- [prometheus-minute-schedule-1,4f0dd5a6e3a5c415,ee69b46c2c8a6990] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:46.521 INFO 26087 --- [ocp-schedules-25,4468f92b472d0356,44741146cca24c42] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:18:46.521 INFO 26087 --- [ocp-schedules-4,222d9dff5a10ee86,38d54d5753b59e6f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:18:46.521 INFO 26087 --- [ocp-schedules-4,222d9dff5a10ee86,38d54d5753b59e6f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:18:46.523 INFO 26087 --- [ocp-schedules-20,a7f2b828303422ab,54cb48cf9a4fc093] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:18:46.523 INFO 26087 --- [ocp-schedules-26,544676db16b3da42,d3fb58bb36734e81] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:18:46.523 INFO 26087 --- [ocp-schedules-20,a7f2b828303422ab,54cb48cf9a4fc093] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:18:46.523 INFO 26087 --- [ocp-schedules-26,544676db16b3da42,d3fb58bb36734e81] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:18:46.524 INFO 26087 --- [ocp-schedules-13,251401ef56a1e25c,8afe00ed178f872d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:18:46.524 INFO 26087 --- [ocp-schedules-23,649a528496abc9a4,a45dee1862acfc13] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:18:46.524 INFO 26087 --- [ocp-schedules-16,a8e630c706e4c26a,65e1e2d34e32ca61] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:18:46.544 INFO 26087 --- [ocp-schedules-11,927baf338b904b9a,6b80f61c848b9bb5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:18:46.552 INFO 26087 --- [ocp-async-4,526610cc9aa5ebe5,3f18feb318f3997a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:18:46.609 INFO 26087 --- [ocp-schedules-28,23b3d19f037f3c29,a225cb60eaf630d8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:18:46.613 WARN 26087 --- [ocp-async-8,b4ee69c0a5dc30e9,a90c6f7fd800a6ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:46.613 INFO 26087 --- [ocp-async-2,b4251f48be2b1941,63b98a9c1a0f4989] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:18:46.622739+08:00, lastProcessTime=2025-04-08T21:18:36.623340+08:00 2025-04-08 21:18:46.665 INFO 26087 --- [ocp-schedules-21,a2eb4c93258ba6fe,657381ad602576a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:18:46.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:18:46.622739+08:00 2025-04-08 21:18:47.340 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:47.340 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:47.993 INFO 26087 --- [ocp-schedules-30,5a5d3feac4f3a88d,743a28b6f462c568] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:18:47.994 INFO 26087 --- [ocp-schedules-30,5a5d3feac4f3a88d,743a28b6f462c568] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:18:48.355 INFO 26087 --- [prometheus-minute-schedule-1,38ddb53c02ce25a3,e109592dd5d591ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:48.613 INFO 26087 --- [ocp-async-9,d56efed8f855e88b,861df5be11a0e80f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:48.613 WARN 26087 --- [ocp-async-6,78935d956e377126,45d504a1fe3f9272] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:49.652 INFO 26087 --- [ocp-schedules-19,d506ff867c03fb87,2f99dd9ae80b362a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:18:50.341 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:50.341 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:50.613 WARN 26087 --- [ocp-async-3,4109f55b6913992a,cd9f42b16cb6e891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:50.613 INFO 26087 --- [ocp-async-1,45301835d4fdfb63,f650c78810019227] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:51.355 INFO 26087 --- [prometheus-minute-schedule-1,38dd9f0a93c305fd,7c0b5e8fbe91c957] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:52.614 WARN 26087 --- [ocp-async-7,ca4d25e55b8facc0,e7293ea3dbba5e53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:52.614 INFO 26087 --- [ocp-async-4,06559b2e01e8039f,385243b817a2dcb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:53.341 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:53.342 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:54.355 INFO 26087 --- [prometheus-minute-schedule-1,987ae941dd2c9440,fa03ab358fc91688] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:54.614 WARN 26087 --- [ocp-async-5,b4dc73a0c96d36f1,d590453aebd484c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:54.614 INFO 26087 --- [ocp-async-6,ae2408ab3d4c540c,1ee6ef0dac015770] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:56.342 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:56.342 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:56.552 INFO 26087 --- [ocp-async-6,a7ce6439498bdaab,38f83c0de0801f53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:18:56.568 INFO 26087 --- [ocp-schedules-23,07af771ad8d10711,13be7e7a3c74e00e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:18:56.614 WARN 26087 --- [ocp-async-3,5d083cd18b5e77d6,4a86925b65078d98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:56.614 INFO 26087 --- [ocp-async-9,1bc3c19db263d43e,ea0658dfa1b3e8d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:18:56.623752+08:00, lastProcessTime=2025-04-08T21:18:46.622739+08:00 2025-04-08 21:18:56.666 INFO 26087 --- [ocp-schedules-11,d82c518dd47f5b76,f0d7a8b13d5754ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:18:56.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:18:56.623752+08:00 2025-04-08 21:18:57.236 INFO 26087 --- [ocp-schedules-29,bd1a6aeb3669e992,4400162a6a613555] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:18:57.241 INFO 26087 --- [ocp-schedules-29,bd1a6aeb3669e992,4400162a6a613555] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:18:57.241 INFO 26087 --- [ocp-schedules-29,bd1a6aeb3669e992,4400162a6a613555] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:18:57.355 INFO 26087 --- [prometheus-minute-schedule-1,14f29f8d3ff0c445,047bc6ff0d41ce3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118340, interval=60 2025-04-08 21:18:57.998 INFO 26087 --- [ocp-schedules-28,9c4c109612718364,eee5ffdd0d934ddd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:18:57.999 INFO 26087 --- [ocp-schedules-28,9c4c109612718364,eee5ffdd0d934ddd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:18:58.614 WARN 26087 --- [ocp-async-7,e07da6f93bf7709c,e6cf8dd35c659a8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:18:58.614 INFO 26087 --- [ocp-async-8,184ebda5aa38d521,2d8dbbeafb59eec3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:18:59.343 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:18:59.343 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:18:59.670 INFO 26087 --- [ocp-schedules-24,f1bc1fdf0d797882,df3686845370c14d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:19:00.355 INFO 26087 --- [prometheus-minute-schedule-1,fe10115477a1f093,91319879774d9722] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:00.614 WARN 26087 --- [ocp-async-5,9921a3ed91adee43,961649f62c3b4cb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:00.614 INFO 26087 --- [ocp-async-2,8674014df849b291,23fea466e95ed45d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:02.343 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:02.344 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:02.614 WARN 26087 --- [ocp-async-10,98eac0fcad08e8e2,c7027e94a5286b2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:02.614 INFO 26087 --- [ocp-async-3,a1945ccf32f543ff,5aaf15a7a438b3a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:03.355 INFO 26087 --- [prometheus-minute-schedule-1,bc51508a3d861751,4efb3ade9a9a868d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:04.614 WARN 26087 --- [ocp-async-4,092b4ec8e73bc699,e4902fd681d139a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:04.614 INFO 26087 --- [ocp-async-9,67c20c6535a81d57,5676c27379d6eace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:05.344 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:05.344 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:06.303 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:13:06.217, end=2025-04-08T21:18:06.282} 2025-04-08 21:19:06.304 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:13:06.228, end=2025-04-08T21:18:06.292} 2025-04-08 21:19:06.307 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:19:06.307 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:19:06.307 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:19:06.307 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:19:06.355 INFO 26087 --- [prometheus-minute-schedule-1,109d5e587cbfa3c1,55c1f66db56b8fec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:06.355 INFO 26087 --- [prometheus-cache-monitor1,338cd201c9e1ab5b,c30384dbf8ee0f65] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1502,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:19:06.355 INFO 26087 --- [prometheus-cache-monitor1,338cd201c9e1ab5b,c30384dbf8ee0f65] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:19:06.552 INFO 26087 --- [ocp-async-9,0aa9844a5c1afbe6,071205ffd83ea29c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:19:06.615 WARN 26087 --- [ocp-async-6,14f5b3376dfa3239,1edc70725e2af836] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:06.615 INFO 26087 --- [ocp-async-5,7fa9c021d07cd310,eb3b47d987088b3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:19:06.623806+08:00, lastProcessTime=2025-04-08T21:18:56.623752+08:00 2025-04-08 21:19:06.667 INFO 26087 --- [ocp-schedules-20,114b0b53ef87ce03,8edd749cb67ea721] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:19:06.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:19:06.623806+08:00 2025-04-08 21:19:08.005 INFO 26087 --- [ocp-schedules-5,68b35f3f3d5f540d,9a6bd9530d517084] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:19:08.006 INFO 26087 --- [ocp-schedules-5,68b35f3f3d5f540d,9a6bd9530d517084] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:19:08.345 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:08.345 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:08.615 WARN 26087 --- [ocp-async-10,48bfa256b351ffcd,916f1856200bd0d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:08.615 INFO 26087 --- [ocp-async-1,45851163acf5c1ba,7d9f00802e923dd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:09.355 INFO 26087 --- [prometheus-minute-schedule-1,a861f947147a7a6b,bfc9adf953fc5d49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:09.692 INFO 26087 --- [ocp-schedules-20,c63bedc8fb0a0ea0,fb1b008552e6c5e4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:19:10.001 INFO 26087 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:19:10.001 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:19:10.604 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:19:10.615 INFO 26087 --- [ocp-async-7,b293bbbebedd9a19,8a6c24949037e0ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:10.615 WARN 26087 --- [ocp-async-4,a8f0d83095f9eede,d1de3857fe241fc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:11.345 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:11.346 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:12.355 INFO 26087 --- [prometheus-minute-schedule-1,4c4782c07b959172,f5e7051bd52d8385] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:12.423 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118280, nodeCount=198, affectRows=198 2025-04-08 21:19:12.423 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118280, nodeCount=180, affectRows=180 2025-04-08 21:19:12.425 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118280, nodeCount=180, validCount=180 2025-04-08 21:19:12.425 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118280, nodeCount=197, validCount=197 2025-04-08 21:19:12.615 WARN 26087 --- [ocp-async-6,1351487d34b881b1,69a9009198c18654] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:12.615 INFO 26087 --- [ocp-async-8,cbb822a77cedbea6,6ba59eabf0a8ffaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:14.346 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:14.346 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:14.615 WARN 26087 --- [ocp-async-10,ddae4050637cdbcd,ff43d029bbb9925f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:14.615 INFO 26087 --- [ocp-async-5,3b5401efb14e8630,5e099fab7849ed03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:15.355 INFO 26087 --- [prometheus-minute-schedule-1,581679fbd7c04539,16db38353e480cce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:16.517 INFO 26087 --- [ocp-schedules-1,b43918a437eea621,eed2a506b872ae3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:19:16.517 INFO 26087 --- [ocp-schedules-20,eb99b0472648ba4c,c093f44490cb2c8b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:19:16.521 INFO 26087 --- [ocp-schedules-19,72ca825687f39895,419520ade1e8db16] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:19:16.521 INFO 26087 --- [ocp-schedules-19,72ca825687f39895,419520ade1e8db16] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:19:16.521 INFO 26087 --- [ocp-schedules-9,2a77c77483bae8a7,d979dba76230661a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:19:16.523 INFO 26087 --- [ocp-schedules-10,937b4652d56cb582,97661d938d6c7c7d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:19:16.523 INFO 26087 --- [ocp-schedules-10,937b4652d56cb582,97661d938d6c7c7d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:19:16.523 INFO 26087 --- [ocp-schedules-19,b884c16928cf61a1,cbf504ed67d59834] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:19:16.523 INFO 26087 --- [ocp-schedules-19,b884c16928cf61a1,cbf504ed67d59834] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:19:16.524 INFO 26087 --- [ocp-schedules-25,221755f7e3bb7946,47087a09f86ee1ff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:19:16.524 INFO 26087 --- [ocp-schedules-18,acd0509f8c1f6b72,08e803885b95e262] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:19:16.524 INFO 26087 --- [ocp-schedules-12,e4b68ebdb80e378b,4f317d2a89a5f61b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:19:16.526 INFO 26087 --- [ocp-schedules-20,eb99b0472648ba4c,c093f44490cb2c8b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:19:16.528 INFO 26087 --- [ocp-schedules-1,b43918a437eea621,eed2a506b872ae3e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:19:16.535 INFO 26087 --- [ocp-schedules-8,5c1f90b9006b50c5,66c0e693e409f932] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:19:16.552 INFO 26087 --- [ocp-async-4,8b035b2e8560c97e,0fdbb66105dc0f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:19:16.560 INFO 26087 --- [ocp-schedules-5,3fcd92d5098d9298,83e1267add7013a7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:19:16.560 INFO 26087 --- [ocp-schedules-5,3fcd92d5098d9298,83e1267add7013a7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:19:16.585 INFO 26087 --- [ocp-schedules-3,a3c744263ef2a6c6,3d494c1125a782af] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1506,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:19:16.585 INFO 26087 --- [ocp-schedules-3,a3c744263ef2a6c6,3d494c1125a782af] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:19:16.591 INFO 26087 --- [ocp-schedules-29,a9dfc8838809dc76,331c29aaac0afd92] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:19:16.611 INFO 26087 --- [ocp-schedules-21,d9c41597ce331eca,79d9df608de758a6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:19:16.615 WARN 26087 --- [ocp-async-6,e9e2442044b53a1f,220edf5a1bcc1c78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:16.615 INFO 26087 --- [ocp-async-2,cf440fa3dcd86744,2974a0736457eb2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:16.622 INFO 26087 --- [ocp-schedules-9,8bca2ab9cc7088e0,f515d9666e232dbf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:19:16.622 INFO 26087 --- [ocp-schedules-9,8bca2ab9cc7088e0,f515d9666e232dbf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:19:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:19:16.622909+08:00, lastProcessTime=2025-04-08T21:19:06.623806+08:00 2025-04-08 21:19:16.642 INFO 26087 --- [ocp-schedules-18,8721cd6641b9e815,6eceff5e736cb47f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:19:16.642 INFO 26087 --- [ocp-schedules-18,8721cd6641b9e815,6eceff5e736cb47f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:19:16.642 INFO 26087 --- [ocp-schedules-10,27379f89a4867894,97a2cd7d7bb4a49e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:19:16.642+08:00, clusters=[] 2025-04-08 21:19:16.643 INFO 26087 --- [ocp-schedules-19,da511ff8ae8e6ab3,c1dd3ffa5a8c644e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:19:16.648 INFO 26087 --- [ocp-schedules-12,dbda4821d45e9131,b7ba3adcdeae2bfc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:19:16.648+08:00, clusters=[] 2025-04-08 21:19:16.667 INFO 26087 --- [ocp-schedules-20,c1580dd00bdb849e,c42c7559e432085f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:19:16.667 INFO 26087 --- [ocp-schedules-20,c1580dd00bdb849e,c42c7559e432085f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:19:16.668 INFO 26087 --- [ocp-schedules-20,c1580dd00bdb849e,c42c7559e432085f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:19:16.668 INFO 26087 --- [ocp-schedules-20,c1580dd00bdb849e,c42c7559e432085f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:19:16.669 INFO 26087 --- [ocp-schedules-8,ff7ef26f07a5cc95,f5b78e0ea8ff500d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:19:16.674 INFO 26087 --- [ocp-schedules-22,87abf9dd115e031b,9ef1bc01d43bdfdf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:19:16.675 INFO 26087 --- [ocp-schedules-22,87abf9dd115e031b,9ef1bc01d43bdfdf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:19:16.675 INFO 26087 --- [ocp-schedules-22,87abf9dd115e031b,9ef1bc01d43bdfdf] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:19:16.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:19:16.622909+08:00 2025-04-08 21:19:16.764 INFO 26087 --- [ocp-schedules-11,6b5ce7112c535b3e,b8d680f2aca78842] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:19:16.825 INFO 26087 --- [ocp-schedules-26,8cac1f50c9d51676,332a22e357a08e36] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:19:16.830 INFO 26087 --- [ocp-schedules-5,746315d9b5d2478b,b08807b8e0ea4a00] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:19:16.838 INFO 26087 --- [ocp-schedules-1,61a26fdc92c73c7b,653c20253c6343b2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:19:16.838 INFO 26087 --- [ocp-schedules-1,61a26fdc92c73c7b,653c20253c6343b2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:19:16.840 INFO 26087 --- [ocp-schedules-28,e02d8e0396c06eb8,5a675e1a51bef3c7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:19:16.841 INFO 26087 --- [ocp-schedules-1,61a26fdc92c73c7b,653c20253c6343b2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:19:16.841 INFO 26087 --- [ocp-schedules-1,61a26fdc92c73c7b,653c20253c6343b2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:19:16.843 INFO 26087 --- [ocp-schedules-1,61a26fdc92c73c7b,653c20253c6343b2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:19:16.843 INFO 26087 --- [ocp-schedules-1,61a26fdc92c73c7b,653c20253c6343b2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:19:17.242 INFO 26087 --- [ocp-schedules-23,f5b684bfa70dca79,b2e60515b28e2c5a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:19:17.252 INFO 26087 --- [ocp-schedules-23,f5b684bfa70dca79,b2e60515b28e2c5a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:19:17.252 INFO 26087 --- [ocp-schedules-23,f5b684bfa70dca79,b2e60515b28e2c5a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:19:17.347 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:17.347 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:17.550 INFO 26087 --- [ocp-schedules-10,8cc730cc276b391b,00619638a2fc333e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:19:17.551 INFO 26087 --- [ocp-schedules-10,8cc730cc276b391b,00619638a2fc333e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:19:18.012 INFO 26087 --- [ocp-schedules-22,e9818a8778c3da7e,58fd9cfecee4abd2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:19:18.014 INFO 26087 --- [ocp-schedules-22,e9818a8778c3da7e,58fd9cfecee4abd2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:19:18.355 INFO 26087 --- [prometheus-minute-schedule-1,d24011b7c93f72f2,02a07efb95ed9ecc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:18.615 WARN 26087 --- [ocp-async-10,0ab658e175d55959,0a3337bfe75713a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:18.615 INFO 26087 --- [ocp-async-3,dd9d0d1dd7917dc4,a48a70112af70238] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:19.709 INFO 26087 --- [ocp-schedules-26,c57b1439ad3f6ca8,159a27613498e63f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:19:20.347 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:20.348 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:19:20.616 WARN 26087 --- [ocp-async-5,c0a20bc3c401b314,ff670469f3fb56c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:20.616 INFO 26087 --- [ocp-async-1,42767ed14c6a7d3b,d862b579d05ba3fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:21.355 INFO 26087 --- [prometheus-minute-schedule-1,aef4bb135f73fd3b,cf6c7023bfb507f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:22.616 WARN 26087 --- [ocp-async-1,d1416b401792420d,ce3c08af9c7e36d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:22.616 INFO 26087 --- [ocp-async-9,28046bdae37d80f4,34038435c1e5e7d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:23.348 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:23.348 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:24.355 INFO 26087 --- [prometheus-minute-schedule-1,63b14d822a69b56a,4169c6562eec4648] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:24.616 WARN 26087 --- [ocp-async-10,20d40bc9ab545b2d,ca5f308ed31f4d0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:24.616 INFO 26087 --- [ocp-async-9,d94e1639acd9551d,050b29a87830ca9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:26.348 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:26.349 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:26.553 INFO 26087 --- [ocp-async-5,cf7947c5fb87f308,7041382369682f46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:19:26.616 WARN 26087 --- [ocp-async-1,0562c50e9f39cce1,50d6a3de7d6b25cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:26.616 INFO 26087 --- [ocp-async-3,edf62b7c56e24345,0b6347e75929a57a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:19:26.623926+08:00, lastProcessTime=2025-04-08T21:19:16.622909+08:00 2025-04-08 21:19:26.670 INFO 26087 --- [ocp-schedules-7,6dcca4a9612f1c72,00fa6e6b36898794] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:19:26.774 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:19:26.623926+08:00 2025-04-08 21:19:27.355 INFO 26087 --- [prometheus-minute-schedule-1,2859819513427f09,2cf09ab3b4b604d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:28.017 INFO 26087 --- [ocp-schedules-20,5d09543f46f41ca5,ba776f3ebfdd06b0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:19:28.018 INFO 26087 --- [ocp-schedules-20,5d09543f46f41ca5,ba776f3ebfdd06b0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:19:28.616 WARN 26087 --- [ocp-async-3,8cb2413f88023e54,dc7c2ad2eff8c414] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:28.616 INFO 26087 --- [ocp-async-6,4468c2ba867abfa1,60bb59d7ebff9842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:29.349 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:29.349 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:29.733 INFO 26087 --- [ocp-schedules-15,31a171f05d34203b,a3d31a3c42889e1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:19:30.355 INFO 26087 --- [prometheus-minute-schedule-1,2ee514c9887098ca,623d3831e378c2b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:30.616 WARN 26087 --- [ocp-async-9,fddd6e0a5c625166,aef38364a576d1d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:30.616 INFO 26087 --- [ocp-async-6,5f7056290e73b65f,5dbf80cbbfa050bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:32.350 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:32.350 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:32.616 WARN 26087 --- [ocp-async-1,c1247356ff9b248b,184b514e7ae20793] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:32.616 INFO 26087 --- [ocp-async-7,4fd4bf752783ce2d,ecd66132c5cbc35a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:33.355 INFO 26087 --- [prometheus-minute-schedule-1,8e68504cff87c60e,cc3752d4649e2ea3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:34.617 WARN 26087 --- [ocp-async-3,f8a44e7d5529aedc,ff7f9943b6deccc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:34.617 INFO 26087 --- [ocp-async-7,bba123a0f79fa4ea,a0a34c770ead0e60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:34.993 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:19:34.993 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:19:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:19:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:19:34.995 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:19:34.996 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:19:35.350 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:35.351 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:36.355 INFO 26087 --- [prometheus-minute-schedule-1,9d00ae9dc11bb959,efa5eafc60ddfb1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:36.553 INFO 26087 --- [ocp-async-9,91ea3bfbadcc406d,6d92cd04d6cb14e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:19:36.617 WARN 26087 --- [ocp-async-8,7d19793ba0cf3029,73f5542782670f02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:36.617 INFO 26087 --- [ocp-async-7,2594b10bdb4f9658,958a0c18ac5fda5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:19:36.622624+08:00, lastProcessTime=2025-04-08T21:19:26.623926+08:00 2025-04-08 21:19:36.671 INFO 26087 --- [ocp-schedules-22,687e47bb832f56f3,7ced35100d4d08d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:19:36.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:19:36.622624+08:00 2025-04-08 21:19:37.253 INFO 26087 --- [ocp-schedules-24,54b74fe7ba9184dd,dbad6de3a0e6e181] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:19:37.259 INFO 26087 --- [ocp-schedules-24,54b74fe7ba9184dd,dbad6de3a0e6e181] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:19:37.259 INFO 26087 --- [ocp-schedules-24,54b74fe7ba9184dd,dbad6de3a0e6e181] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:19:38.022 INFO 26087 --- [ocp-schedules-16,a35864e85151a122,80d861fd433cb2c2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:19:38.023 INFO 26087 --- [ocp-schedules-16,a35864e85151a122,80d861fd433cb2c2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:19:38.351 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:38.351 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:38.617 WARN 26087 --- [ocp-async-4,a25bc8ce6a057500,0772ea23c1a957cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:38.617 INFO 26087 --- [ocp-async-1,81a5a1c300c86446,f0df7ef19382c71e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:39.355 INFO 26087 --- [prometheus-minute-schedule-1,745c5988ab25951f,5a6d60efba9a95fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:39.749 INFO 26087 --- [ocp-schedules-16,1d8ef9aea34aaf6f,22f8607014e0aa32] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:19:40.001 INFO 26087 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:19:40.002 INFO 26087 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:19:40.617 WARN 26087 --- [ocp-async-3,b3edbebb52162ca9,7e0a1cc273cc3628] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:40.617 INFO 26087 --- [ocp-async-2,f3c810eddcd1d836,0759f3541580669f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:41.351 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:41.352 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:42.355 INFO 26087 --- [prometheus-minute-schedule-1,f1dcdcc91e634d74,6599f726c541b322] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:42.617 WARN 26087 --- [ocp-async-9,b476f1fa8830101f,7bb9ba35838d3e48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:42.617 INFO 26087 --- [ocp-async-5,9cae6812f408d031,564189adac11672f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:44.352 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:44.352 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:44.617 WARN 26087 --- [ocp-async-4,49395766bf4d92d8,f168f850a1a4162c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:44.617 INFO 26087 --- [ocp-async-5,68b9e7de61336c02,07320dbde2a3f147] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:45.355 INFO 26087 --- [prometheus-minute-schedule-1,e8f525ea4e12671c,c5f94754c9d4e366] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:46.521 INFO 26087 --- [ocp-schedules-25,5a96731987a20eb1,bba205b49df0f5cc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:19:46.521 INFO 26087 --- [ocp-schedules-22,7db6bde09826ad12,fd4d9a5b043c551d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:19:46.521 INFO 26087 --- [ocp-schedules-22,7db6bde09826ad12,fd4d9a5b043c551d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:19:46.523 INFO 26087 --- [ocp-schedules-27,5ad47b22ce197b64,beaa03efd5ad6d2d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:19:46.523 INFO 26087 --- [ocp-schedules-10,9b20f26d49f1b1fa,e88d1b71ba6cfa25] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:19:46.523 INFO 26087 --- [ocp-schedules-27,5ad47b22ce197b64,beaa03efd5ad6d2d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:19:46.523 INFO 26087 --- [ocp-schedules-10,9b20f26d49f1b1fa,e88d1b71ba6cfa25] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:19:46.524 INFO 26087 --- [ocp-schedules-6,1c7a94182ece1b06,24b6cca967802a4f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:19:46.524 INFO 26087 --- [ocp-schedules-9,a0f08a6be0c298fa,c92617bc5ee1759d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:19:46.524 INFO 26087 --- [ocp-schedules-3,29b122e7efb63995,34731a9fff48ff20] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:19:46.533 INFO 26087 --- [ocp-schedules-26,d0b8d958a2281a42,f948bed954d25ae3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:19:46.553 INFO 26087 --- [ocp-async-6,28d3dc05091b69c9,2a44bbea104d01b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:19:46.612 INFO 26087 --- [ocp-schedules-4,68cbb7a99f5119f6,91a33fe40a9886c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:19:46.617 INFO 26087 --- [ocp-async-3,5c3df7127121e5ad,8ce0f2442c2c15b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:46.617 WARN 26087 --- [ocp-async-2,7107ad462d3e194e,1d681181cbfec179] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:19:46.623696+08:00, lastProcessTime=2025-04-08T21:19:36.622624+08:00 2025-04-08 21:19:46.672 INFO 26087 --- [ocp-schedules-20,909b8edbc2459a54,34f814653afabbf4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:19:46.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:19:46.623696+08:00 2025-04-08 21:19:47.353 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:47.353 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:48.027 INFO 26087 --- [ocp-schedules-9,60c709bf9b6f04a6,7cff5eba7c08dfce] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:19:48.028 INFO 26087 --- [ocp-schedules-9,60c709bf9b6f04a6,7cff5eba7c08dfce] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:19:48.355 INFO 26087 --- [prometheus-minute-schedule-1,2bd4f8b4615375da,4b743c2f2672be03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:48.617 WARN 26087 --- [ocp-async-8,e497b71155e98377,a6aff079d8691142] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:48.617 INFO 26087 --- [ocp-async-9,79a3bd5acc0ad9a9,fecabd92dd4fd2ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:49.768 INFO 26087 --- [ocp-schedules-27,2c97a30e17e4b5c9,eef95c31b3e18b4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:19:50.353 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:50.354 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:50.618 WARN 26087 --- [ocp-async-5,7c1d3be3edb4eed3,f36fbf8d2b9dd85b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:50.618 INFO 26087 --- [ocp-async-4,ebef742f13f2938d,75bd6b84c901f65b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:51.355 INFO 26087 --- [prometheus-minute-schedule-1,8c2b411ae2f1e973,3a9f80f3722ffe20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:52.618 WARN 26087 --- [ocp-async-10,92bef193ff6831b0,08a82bf45470246f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:52.618 INFO 26087 --- [ocp-async-2,4a67878d0f7706bd,8c0afa8a0e156d07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:53.354 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:53.354 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:54.355 INFO 26087 --- [prometheus-minute-schedule-1,1a907e9101a8dab0,01d6056a5b5343e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:54.618 WARN 26087 --- [ocp-async-8,1e41a4c424acb07d,c38c0c9c7118dc70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:54.618 INFO 26087 --- [ocp-async-3,00f2e2eeff1e00c4,307b8c8e82b19e6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:56.355 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:56.355 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:56.553 INFO 26087 --- [ocp-async-3,0d46e662612de7ad,1385525743cad519] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:19:56.570 INFO 26087 --- [ocp-schedules-22,9797228695d4b1c0,dbcfa2c106b5bbb4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:19:56.618 WARN 26087 --- [ocp-async-1,5bf86d4f7bb5f464,32c16394a5697d3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:56.618 INFO 26087 --- [ocp-async-5,cf5c989b5be6cb4e,4a6f1556bb2ec418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:19:56.621956+08:00, lastProcessTime=2025-04-08T21:19:46.623696+08:00 2025-04-08 21:19:56.673 INFO 26087 --- [ocp-schedules-3,6c1d05f7cf77eaca,05305cd8c7341b95] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:19:56.727 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:19:56.621956+08:00 2025-04-08 21:19:57.261 INFO 26087 --- [ocp-schedules-21,836d77424786e0de,cdf5b76cd94676ac] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:19:57.265 INFO 26087 --- [ocp-schedules-21,836d77424786e0de,cdf5b76cd94676ac] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:19:57.265 INFO 26087 --- [ocp-schedules-21,836d77424786e0de,cdf5b76cd94676ac] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:19:57.355 INFO 26087 --- [prometheus-minute-schedule-1,fe5c6b339f2c7965,cd92cbc7a315df3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118400, interval=60 2025-04-08 21:19:58.032 INFO 26087 --- [ocp-schedules-25,b69b639ea5a495b6,ad1d10a85a03bb12] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:19:58.033 INFO 26087 --- [ocp-schedules-25,b69b639ea5a495b6,ad1d10a85a03bb12] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:19:58.618 WARN 26087 --- [ocp-async-5,d868e82a15ffb79f,9ff934fdaa82a4e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:19:58.618 INFO 26087 --- [ocp-async-2,f0470349a4d000d6,7edd66f0d4682915] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:19:59.355 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:19:59.356 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:19:59.786 INFO 26087 --- [ocp-schedules-10,735dda75ff3b5e9c,712e162357387aa3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:20:00.345 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:15:00.0, nextRun=2025-04-08 21:20:00.0, currentDate=Tue Apr 08 21:20:00 CST 2025 2025-04-08 21:20:00.346 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:15:00.0, nextRunTime=2025-04-08 21:20:00.0} 2025-04-08 21:20:00.349 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:20:00.349 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:20:00.349+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:20:00.355 INFO 26087 --- [prometheus-minute-schedule-1,71571e92d72b51a6,ebdd601215d93015] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:00.359 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:15:00.0, nextRun=2025-04-08 21:20:00.0, currentDate=Tue Apr 08 21:20:00 CST 2025 2025-04-08 21:20:00.359 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:15:00.0, nextRunTime=2025-04-08 21:20:00.0} 2025-04-08 21:20:00.360 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:20:00.362 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:20:00.362 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,32b3b5fff893984f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:20:00.390 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,14866d27aa8bc8f9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000120, operation:EXECUTE, state: RUNNING 2025-04-08 21:20:00.413 INFO 26087 --- [subtask-executor8,8ec1a44aeefd3a62,145d8165fcf10fd8] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:20:00.414 INFO 26087 --- [subtask-executor8,8ec1a44aeefd3a62,145d8165fcf10fd8] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000120, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000011, task_operation=execute, latest_execution_start_time=2025-04-08T21:20:00.386+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000120}, listMap={}}, executor=192.168.1.61 2025-04-08 21:20:00.424 INFO 26087 --- [subtask-executor8,8ec1a44aeefd3a62,145d8165fcf10fd8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000120, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:20:00.618 WARN 26087 --- [ocp-async-7,8de6c4266d9bf2b3,273e706678c8a6f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:00.618 INFO 26087 --- [ocp-async-8,c06a03166a8743d4,e78644f29f25ce2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:01.078 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000011, status:SUCCESS 2025-04-08 21:20:01.078 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000011, status:SUCCESS 2025-04-08 21:20:02.356 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:02.356 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:02.618 WARN 26087 --- [ocp-async-10,09d50e6f6cecdfd5,6c14446617cb0ca0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:02.618 INFO 26087 --- [ocp-async-1,7e351ca7ab594491,b46324fe41e52605] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:03.355 INFO 26087 --- [prometheus-minute-schedule-1,d9818846d028230d,410d7f0b0ccf8c38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:04.440 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,14aadbb60060b028] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:10:05.0, nextRun=2025-04-08 21:20:04.0, currentDate=Tue Apr 08 21:20:04 CST 2025 2025-04-08 21:20:04.440 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,14aadbb60060b028] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-04-08 21:10:05.0, nextRunTime=2025-04-08 21:20:04.0} 2025-04-08 21:20:04.442 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,14aadbb60060b028] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 21:20:04.443 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,14aadbb60060b028] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:20:04.442+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:20:04.619 WARN 26087 --- [ocp-async-3,d6268692034f7ea9,3e4893655eb0d4ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:04.619 INFO 26087 --- [ocp-async-6,5725dabdee65ab07,4303624604da33bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:05.357 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:05.357 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:05.417 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,f88a88acad8b3c28] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000121, operation:EXECUTE, state: RUNNING 2025-04-08 21:20:05.429 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:20:05.431 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000121, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000012, task_operation=execute, latest_execution_start_time=2025-04-08T21:20:05.416+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=1000121}, listMap={}}, executor=192.168.1.61 2025-04-08 21:20:05.436 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:20:05.436 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:20:05.437 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.437 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:20:05.438 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.438 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.439 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.439 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:20:05.452 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.452 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.453 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.453 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:20:05.464 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.464 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.464 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.465 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:20:05.475 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.475 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.476 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.476 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:20:05.487 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.487 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.487 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.487 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:20:05.492 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.493 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.493 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.493 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:20:05.496 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.497 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.497 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.497 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:20:05.501 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.501 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.501 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.502 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:20:05.506 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.507 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.507 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.507 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:20:05.513 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.514 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.514 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.514 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:20:05.519 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.519 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.519 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.520 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:20:05.526 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.527 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.527 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.527 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:20:05.538 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.539 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.539 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.539 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:20:05.543 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.543 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.543 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.544 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:20:05.550 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.550 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.551 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.551 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:20:05.555 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.555 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.555 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.556 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:20:05.559 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.559 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.559 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.559 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:20:05.563 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.563 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.563 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.563 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:20:05.566 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.567 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.567 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.567 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:20:05.571 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.571 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.572 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.572 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:20:05.576 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.576 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.576 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.576 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:20:05.579 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.580 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.580 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.580 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:20:05.583 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.584 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.584 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.584 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:20:05.587 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.587 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.588 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.588 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:20:05.591 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.591 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.591 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.592 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:20:05.595 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.595 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.595 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.596 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:20:05.599 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.599 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.599 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.599 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:20:05.603 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.603 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.603 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.604 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:20:05.607 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.607 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.607 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.607 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:20:05.611 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.611 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.611 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.611 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:20:05.614 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.614 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.615 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.615 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:20:05.618 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.618 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.619 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.619 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:20:05.623 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.623 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.623 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.623 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:20:05.627 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.627 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.627 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.627 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:20:05.633 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.633 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.633 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.634 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:20:05.638 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.639 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.639 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.639 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:20:05.643 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.643 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.644 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.644 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:20:05.648 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.648 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.649 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.649 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:20:05.651 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.652 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.652 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.652 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:20:05.655 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.655 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.656 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.656 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:20:05.658 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.659 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.659 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.659 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:20:05.661 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.662 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.662 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.662 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:20:05.670 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.670 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.670 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.671 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:20:05.687 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.687 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.688 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.688 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:20:05.692 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.693 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.693 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.693 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:20:05.698 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.699 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.699 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.699 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:20:05.704 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.704 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.704 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.705 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:20:05.708 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.708 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.708 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.708 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:20:05.712 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.712 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.712 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.712 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:20:05.715 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:20:05.715 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:20:05.716 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.716 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:20:05.717 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.717 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.717 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.718 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:20:05.721 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.721 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:20:05.721 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:20:05.722 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:20:05.724 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:20:05.743 INFO 26087 --- [subtask-executor9,8ec1a44aeefd3a62,ee8f50597f20ea42] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000121, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:20:06.103 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000012, status:SUCCESS 2025-04-08 21:20:06.103 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000012, status:SUCCESS 2025-04-08 21:20:06.355 INFO 26087 --- [prometheus-minute-schedule-1,73ace7855629443f,790e1d0ca2df6380] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:06.355 INFO 26087 --- [prometheus-cache-monitor1,e18d3de95e0eaf32,4d84f6887d67607e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1506,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:20:06.355 INFO 26087 --- [prometheus-cache-monitor1,e18d3de95e0eaf32,4d84f6887d67607e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:20:06.444 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:16:06.406, end=2025-04-08T21:18:06.44} 2025-04-08 21:20:06.446 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:20:06.446 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:20:06.553 INFO 26087 --- [ocp-async-6,ffbb4f8ff48eda3a,16eecc757d8fd350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:20:06.619 INFO 26087 --- [ocp-async-7,da4d256f8dd48b7f,fe5ad0c2fff9a467] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:06.619 WARN 26087 --- [ocp-async-8,4fdf5b7c1ccf42ee,5b135c18c8c338f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:20:06.624012+08:00, lastProcessTime=2025-04-08T21:19:56.621956+08:00 2025-04-08 21:20:06.674 INFO 26087 --- [ocp-schedules-18,d4cd553a6a4c9699,dce47a54d7cac090] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:20:06.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:20:06.624012+08:00 2025-04-08 21:20:08.038 INFO 26087 --- [ocp-schedules-29,75c5d033f2f022aa,995d65bb2cc886ea] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:20:08.039 INFO 26087 --- [ocp-schedules-29,75c5d033f2f022aa,995d65bb2cc886ea] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:20:08.357 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:08.358 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:08.619 WARN 26087 --- [ocp-async-4,3429998ae7ef6a38,76ceb4eb56a093b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:08.619 INFO 26087 --- [ocp-async-10,3d87678634f3b0de,144466668d16fd1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:09.355 INFO 26087 --- [prometheus-minute-schedule-1,f403c56164297776,6ebde2a0a3ed9c16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:09.806 INFO 26087 --- [ocp-schedules-17,7f14642b676302ab,04cd554550a011d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:20:10.001 INFO 26087 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:20:10.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:20:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:20:10.619 WARN 26087 --- [ocp-async-3,3fbabcf28fb321b0,5f98c1be7f557d68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:10.619 INFO 26087 --- [ocp-async-5,1fa429b93bf5d123,d025fabaef927fb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:11.358 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:11.358 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:12.355 INFO 26087 --- [prometheus-minute-schedule-1,77368e5f2ff75318,b879b1e0a0091efe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:12.421 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118340, nodeCount=378, affectRows=378 2025-04-08 21:20:12.423 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118340, nodeCount=377, validCount=377 2025-04-08 21:20:12.619 WARN 26087 --- [ocp-async-8,9a371f8b0febd9d5,8ab3fb5d5b6f9e0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:12.619 INFO 26087 --- [ocp-async-5,4b9dc7f9325d202e,8f15e60abd6ae343] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:14.359 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:14.359 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:14.619 WARN 26087 --- [ocp-async-7,e9671a19cb2e3fc7,517eba33dcae97b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:14.619 INFO 26087 --- [ocp-async-1,6f6250f6d975b2ad,659d36f02cd4180a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:15.355 INFO 26087 --- [prometheus-minute-schedule-1,4b7452e4d144dd86,b4816f2cb9bcd40f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:16.515 INFO 26087 --- [ocp-schedules-12,a311b3bed356738f,5621a42cd144342c] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:20:16.515+08:00, tenantSize=0 2025-04-08 21:20:16.517 INFO 26087 --- [ocp-schedules-20,47329eae5a260f4d,7da1ddb18fd25fae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:20:16.517 INFO 26087 --- [ocp-schedules-2,6ea827037f8935bf,24742a1fc45a12a9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:20:16.521 INFO 26087 --- [ocp-schedules-7,a6ba3d8a35a3cff4,34dc698b4108bbd6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:20:16.521 INFO 26087 --- [ocp-schedules-7,a6ba3d8a35a3cff4,34dc698b4108bbd6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:20:16.521 INFO 26087 --- [ocp-schedules-28,22b414c0810d63d1,e3be430b1620ac7d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:20:16.523 INFO 26087 --- [ocp-schedules-23,bc9747fb98c9a010,e2fa498db30f337f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:20:16.523 INFO 26087 --- [ocp-schedules-23,bc9747fb98c9a010,e2fa498db30f337f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:20:16.524 INFO 26087 --- [ocp-schedules-17,0ec68425753a1c39,bedaf71e5b732f3b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:20:16.524 INFO 26087 --- [ocp-schedules-17,0ec68425753a1c39,bedaf71e5b732f3b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:20:16.524 INFO 26087 --- [ocp-schedules-8,14713385014b1f1f,e12bb5b49044f863] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:20:16.524 INFO 26087 --- [ocp-schedules-4,609d27d00e6765b9,2c6bf83957593f35] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:20:16.524 INFO 26087 --- [ocp-schedules-1,81ad44885959f8c9,f194c51d7804b5b8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:20:16.525 INFO 26087 --- [ocp-schedules-2,6ea827037f8935bf,24742a1fc45a12a9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:20:16.553 INFO 26087 --- [ocp-async-3,4a1fe4d0ea432f1d,cc397f1fcbf567eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:20:16.592 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:20:16.592 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:20:16.593 INFO 26087 --- [ocp-schedules-30,a6132f983f972443,7a2f3f9271cd0921] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:20:16.596 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:20:16.596 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:20:16.596 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:20:16.598 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:20:16.598 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:20:16.598 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:20:16.599 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:20:16.599 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:20:16.599 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:20:16.601 INFO 26087 --- [ocp-schedules-27,9778d07b4351e40b,3138a49b3e15602b] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:20:16.604 INFO 26087 --- [ocp-schedules-20,47329eae5a260f4d,7da1ddb18fd25fae] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:20:16.613 INFO 26087 --- [ocp-schedules-5,51ed7d94da2f03bd,af19b086df83886f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:20:16.615 INFO 26087 --- [ocp-schedules-24,1f6e0e5f6d19f4db,0b4e79fbe316c7ff] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:20:16.620 INFO 26087 --- [ocp-async-6,f7f6e00b6fd300f5,38ff64245e2efded] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:16.620 WARN 26087 --- [ocp-async-9,efa2b4621ef754d8,88bf877d64a6037b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:20:16.623696+08:00, lastProcessTime=2025-04-08T21:20:06.624012+08:00 2025-04-08 21:20:16.624 INFO 26087 --- [ocp-schedules-7,0566077a7e1095b7,3f4a369f1de0ddc4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:20:16.624 INFO 26087 --- [ocp-schedules-7,0566077a7e1095b7,3f4a369f1de0ddc4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:20:16.643 INFO 26087 --- [ocp-schedules-17,082e471e1952d6f8,5112fb15c3484a6a] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:20:16.643 INFO 26087 --- [ocp-schedules-17,082e471e1952d6f8,5112fb15c3484a6a] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:20:16.643 INFO 26087 --- [ocp-schedules-28,665f81d985b95732,013c43bec47fcced] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:20:16.643+08:00, clusters=[] 2025-04-08 21:20:16.644 INFO 26087 --- [ocp-schedules-23,d41c69fd248e1cb3,2eead507c7898deb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:20:16.649 INFO 26087 --- [ocp-schedules-4,ed37f07739763ccb,3c7ceab767c7a3a8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:20:16.649+08:00, clusters=[] 2025-04-08 21:20:16.668 INFO 26087 --- [ocp-schedules-1,16edf75583ac441b,b03731f292abee09] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:20:16.668 INFO 26087 --- [ocp-schedules-1,16edf75583ac441b,b03731f292abee09] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:20:16.669 INFO 26087 --- [ocp-schedules-1,16edf75583ac441b,b03731f292abee09] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:20:16.669 INFO 26087 --- [ocp-schedules-1,16edf75583ac441b,b03731f292abee09] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:20:16.675 INFO 26087 --- [ocp-schedules-8,a2c65e5cb790f2c3,8fbea1c1b02d5e72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:20:16.676 INFO 26087 --- [ocp-schedules-22,a33d50c68b37a973,7e5fb99f23e2415a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:20:16.676 INFO 26087 --- [ocp-schedules-8,a2c65e5cb790f2c3,8fbea1c1b02d5e72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:20:16.676 INFO 26087 --- [ocp-schedules-8,a2c65e5cb790f2c3,8fbea1c1b02d5e72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:20:16.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:20:16.623696+08:00 2025-04-08 21:20:16.771 INFO 26087 --- [ocp-schedules-14,a57a4e6ed4679095,ba0d2c08d75fb116] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:20:16.826 INFO 26087 --- [ocp-schedules-11,76ef93e2719534c5,f86de5c2f3ff2972] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:20:16.840 INFO 26087 --- [ocp-schedules-16,d3e2e84cbf80124f,65cd2ddd393397ec] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:20:16.845 INFO 26087 --- [ocp-schedules-13,e56eed7d7f201eb6,772e45be8fd896cc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:20:16.845 INFO 26087 --- [ocp-schedules-13,e56eed7d7f201eb6,772e45be8fd896cc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:20:16.846 INFO 26087 --- [ocp-schedules-13,e56eed7d7f201eb6,772e45be8fd896cc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:20:16.846 INFO 26087 --- [ocp-schedules-13,e56eed7d7f201eb6,772e45be8fd896cc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:20:16.846 INFO 26087 --- [ocp-schedules-13,e56eed7d7f201eb6,772e45be8fd896cc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:20:16.846 INFO 26087 --- [ocp-schedules-13,e56eed7d7f201eb6,772e45be8fd896cc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:20:17.266 INFO 26087 --- [ocp-schedules-5,3c8b231962c444d8,d7ca8e2d9d24dccb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:20:17.271 INFO 26087 --- [ocp-schedules-5,3c8b231962c444d8,d7ca8e2d9d24dccb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:20:17.271 INFO 26087 --- [ocp-schedules-5,3c8b231962c444d8,d7ca8e2d9d24dccb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:20:17.359 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:17.360 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:17.554 INFO 26087 --- [ocp-schedules-20,e0a22eac88efe01a,b24a01a9be98c549] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:20:17.554 INFO 26087 --- [ocp-schedules-20,e0a22eac88efe01a,b24a01a9be98c549] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:20:18.045 INFO 26087 --- [ocp-schedules-29,2634bb5fc3ef0ef3,2d5fd73101081b2e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:20:18.046 INFO 26087 --- [ocp-schedules-29,2634bb5fc3ef0ef3,2d5fd73101081b2e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:20:18.355 INFO 26087 --- [prometheus-minute-schedule-1,048b4e2ce232f64c,13f2cc57b2bd8543] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:18.620 WARN 26087 --- [ocp-async-4,fee1aaa12ad39619,ac153f95977605bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:18.620 INFO 26087 --- [ocp-async-7,bb6fd32e01c38ba7,04471aae84de2288] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:19.825 INFO 26087 --- [ocp-schedules-8,bdf44f4e9d113d51,2f8baed4f50b8e6e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:20:20.360 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:20.360 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:20:20.620 WARN 26087 --- [ocp-async-10,e10001de87a5a688,7100275fe0b15759] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:20.620 INFO 26087 --- [ocp-async-1,3ccaf43eddaaab26,9d22a9a1682e049d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:21.355 INFO 26087 --- [prometheus-minute-schedule-1,e69ddc6c3afe5c06,ad3c33b764907f17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:22.620 WARN 26087 --- [ocp-async-9,03760ff4a3b38118,34ce136d81d3a1ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:22.620 INFO 26087 --- [ocp-async-8,faa513d3302c24fa,ead655c2b8419219] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:23.361 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:23.361 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:24.355 INFO 26087 --- [prometheus-minute-schedule-1,bdcb67219157c479,e44b06a4447512a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:24.620 WARN 26087 --- [ocp-async-6,9c89fbb86008c7d9,17e8658552845664] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:24.620 INFO 26087 --- [ocp-async-4,bf46dfda06a74500,c58d620aa6a3c99f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:26.361 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:26.362 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:26.553 INFO 26087 --- [ocp-async-7,3df426e269068844,0c07fd432512b3ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:20:26.620 WARN 26087 --- [ocp-async-10,099e6c50d12050b4,e4b876f88f81028c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:26.620 INFO 26087 --- [ocp-async-1,6d4c6c50fc7becfb,01361cecedbd9dd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:20:26.622747+08:00, lastProcessTime=2025-04-08T21:20:16.623696+08:00 2025-04-08 21:20:26.677 INFO 26087 --- [ocp-schedules-8,f8cbd2e2181225f8,096108e8960f60fd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:20:26.725 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:20:26.622747+08:00 2025-04-08 21:20:27.355 INFO 26087 --- [prometheus-minute-schedule-1,1c74b2a1dee80321,7c7ff822bbf0b485] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:28.050 INFO 26087 --- [ocp-schedules-18,8bc2e1825b2e3fb8,62657076ae7e5715] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:20:28.051 INFO 26087 --- [ocp-schedules-18,8bc2e1825b2e3fb8,62657076ae7e5715] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:20:28.620 WARN 26087 --- [ocp-async-8,ebdbe9b477515682,4ca9f850d0783a2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:28.620 INFO 26087 --- [ocp-async-3,a4a0662d49be2472,b53b2c9821c756e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:29.362 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:29.363 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:29.847 INFO 26087 --- [ocp-schedules-10,6abf1477c8868b99,f8a261febc8f4550] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:20:30.355 INFO 26087 --- [prometheus-minute-schedule-1,bf18886bffa85736,5bd170d08b168420] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:30.620 WARN 26087 --- [ocp-async-4,9879a62c2673eed7,6a7de9c531d6e567] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:30.621 INFO 26087 --- [ocp-async-3,790d7a45d1b7be12,f6e1a0c64e7fcf25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:32.363 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:32.363 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:32.621 WARN 26087 --- [ocp-async-10,c48a73e55a26be2a,893f23adcfb9fbb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:32.621 INFO 26087 --- [ocp-async-3,04018445eff6f24d,5db318ece35da176] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:33.355 INFO 26087 --- [prometheus-minute-schedule-1,9694ef7c5c9ebddc,bb32d63551416f29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:34.621 WARN 26087 --- [ocp-async-8,53165114d603bb23,7702b88fd909ca72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:34.621 INFO 26087 --- [ocp-async-1,705512af0b61f083,5ca361d6b94f4654] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:35.363 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:35.364 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:36.355 INFO 26087 --- [prometheus-minute-schedule-1,d7410ccea056649d,4e85bea007f76dde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:36.554 INFO 26087 --- [ocp-async-1,d9966e1d9d566057,ea8f4cd055f3a024] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:20:36.621 WARN 26087 --- [ocp-async-4,1fa6ec93916a7c89,6f9dee57f1794b02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:36.621 INFO 26087 --- [ocp-async-5,5f1ae85aba765b44,cf250638e05c10f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:20:36.622692+08:00, lastProcessTime=2025-04-08T21:20:26.622747+08:00 2025-04-08 21:20:36.678 INFO 26087 --- [ocp-schedules-13,dc0496e323cc572f,a495b22ed90efab1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:20:36.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:20:36.622692+08:00 2025-04-08 21:20:37.273 INFO 26087 --- [ocp-schedules-17,ce255057d02dae43,c9a0af1a085ccb97] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:20:37.281 INFO 26087 --- [ocp-schedules-17,ce255057d02dae43,c9a0af1a085ccb97] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:20:37.281 INFO 26087 --- [ocp-schedules-17,ce255057d02dae43,c9a0af1a085ccb97] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:20:38.059 INFO 26087 --- [ocp-schedules-18,1eca46c3851581f7,2ff1bd16d9a885d9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:20:38.060 INFO 26087 --- [ocp-schedules-18,1eca46c3851581f7,2ff1bd16d9a885d9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:20:38.364 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:38.365 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:38.621 WARN 26087 --- [ocp-async-5,4dbb008d335240ab,4fae2d25749ef73c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:38.621 INFO 26087 --- [ocp-async-2,d0ec46fdf9ddc178,bb7b1e2404bcf2a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:39.355 INFO 26087 --- [prometheus-minute-schedule-1,2c0720487f259aa0,5ef61d39e2c22628] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:39.865 INFO 26087 --- [ocp-schedules-27,d5fdf07dad69df83,e5cf17c9dbaee1db] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:20:40.001 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:20:40.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:20:40.621 WARN 26087 --- [ocp-async-3,b479b4bfaf14c146,4c1c993db7aee635] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:40.621 INFO 26087 --- [ocp-async-6,db4a8348970dcd6c,fd38db718cb09561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:41.365 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:41.365 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:42.355 INFO 26087 --- [prometheus-minute-schedule-1,b44483b2a6839e42,fdbf2ac8561247ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:42.621 WARN 26087 --- [ocp-async-4,8b2ef7aac9e13b02,1fbba44b952385ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:42.621 INFO 26087 --- [ocp-async-7,395ea5803a2fa357,fe7f37b774ba3b2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:44.365 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:44.366 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:44.622 WARN 26087 --- [ocp-async-10,9652ff21bccccdb4,65b9fffd6decc8c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:44.622 INFO 26087 --- [ocp-async-9,26db85f33c0d1abb,233d5f0b2621f751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:45.355 INFO 26087 --- [prometheus-minute-schedule-1,3af0b36b15eb3f84,aea0ce9121c13138] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:46.521 INFO 26087 --- [ocp-schedules-16,fe36686cd4632de1,a1d83d67cb017d49] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:20:46.521 INFO 26087 --- [ocp-schedules-28,b43d6611d8b45d47,0fa1436bcb85925e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:20:46.521 INFO 26087 --- [ocp-schedules-28,b43d6611d8b45d47,0fa1436bcb85925e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:20:46.523 INFO 26087 --- [ocp-schedules-9,6629289a6967954b,040f30569a567bc3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:20:46.523 INFO 26087 --- [ocp-schedules-9,6629289a6967954b,040f30569a567bc3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:20:46.524 INFO 26087 --- [ocp-schedules-25,40211a9894cf2273,cc7f8ca7f323dda4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:20:46.524 INFO 26087 --- [ocp-schedules-25,40211a9894cf2273,cc7f8ca7f323dda4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:20:46.524 INFO 26087 --- [ocp-schedules-13,458edea8f8938ae0,bcb9e04eb6729730] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:20:46.524 INFO 26087 --- [ocp-schedules-5,c59accdb5199890d,5f92d76288f899fd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:20:46.524 INFO 26087 --- [ocp-schedules-6,9b6ee026b2ccd073,43e1b390ee5c63ef] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:20:46.536 INFO 26087 --- [ocp-schedules-18,a7520fe487b69d3d,13eaaadb7c08a4e4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:20:46.554 INFO 26087 --- [ocp-async-9,4c0a02bcb08b6f81,58b6d1590d96943f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:20:46.614 INFO 26087 --- [ocp-schedules-26,77997d6cf27b5178,3448ab5085e11f80] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:20:46.622 WARN 26087 --- [ocp-async-1,a0ce5b5044d44541,5e917ff597cb5b43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:46.622 INFO 26087 --- [ocp-async-2,d08ddd061385d434,fee65224b53a6079] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:20:46.622860+08:00, lastProcessTime=2025-04-08T21:20:36.622692+08:00 2025-04-08 21:20:46.679 INFO 26087 --- [ocp-schedules-20,72a432ddd9c9509c,8b87bfff9a0dc5b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:20:46.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:20:46.622860+08:00 2025-04-08 21:20:47.366 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:47.367 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:48.066 INFO 26087 --- [ocp-schedules-3,e4fd78746a3bbf96,d90a7b80ce14c957] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:20:48.068 INFO 26087 --- [ocp-schedules-3,e4fd78746a3bbf96,d90a7b80ce14c957] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:20:48.355 INFO 26087 --- [prometheus-minute-schedule-1,a4923a5e6c29329f,d704f68bcc416469] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:48.622 WARN 26087 --- [ocp-async-7,0006e022d6ddc1d5,a944fddc491762d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:48.622 INFO 26087 --- [ocp-async-6,59e9d540f50c5daf,1499a7bf4e1d6c66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:49.884 INFO 26087 --- [ocp-schedules-8,14d37720352f2cbc,e98d320e39875819] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:20:50.367 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:50.367 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:50.622 WARN 26087 --- [ocp-async-8,74c8c21657feac09,7986036c4ffefc77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:50.622 INFO 26087 --- [ocp-async-10,2c166fef0794d855,74ab17ea950c33c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:51.355 INFO 26087 --- [prometheus-minute-schedule-1,2c4f8ebfcb0ca4e2,3a4353aa7265c40e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:52.622 WARN 26087 --- [ocp-async-1,75711837456b71c1,31121800f7372df8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:52.622 INFO 26087 --- [ocp-async-3,f342f60fd59f94dd,809f5b67704c1296] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:53.367 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:53.368 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:54.355 INFO 26087 --- [prometheus-minute-schedule-1,955882b8b09f1753,474302f001d2d82d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:54.622 WARN 26087 --- [ocp-async-7,a08d76af09f95646,8e1c0fd70d4f2324] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:54.622 INFO 26087 --- [ocp-async-2,a6d1ab9e39a50a07,6b17cc3b916a23f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:56.368 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:56.369 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:56.554 INFO 26087 --- [ocp-async-6,4dd114bfc6a192c6,49b2ec62415bbb5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:20:56.570 INFO 26087 --- [ocp-schedules-29,b72e091a8a0cb6ff,77550ad02e572f8e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:20:56.622 WARN 26087 --- [ocp-async-8,9d2724036eb3ddc5,a7bbf31eef81694d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:56.622 INFO 26087 --- [ocp-async-8,7490c027863ec8f1,7f92f22dec937276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:20:56.624163+08:00, lastProcessTime=2025-04-08T21:20:46.622860+08:00 2025-04-08 21:20:56.681 INFO 26087 --- [ocp-schedules-15,767405131b2ad7fe,6e6a3b324a5d10ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:20:56.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:20:56.624163+08:00 2025-04-08 21:20:57.282 INFO 26087 --- [ocp-schedules-4,6138833039791592,080f2cf3a510e92f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:20:57.286 INFO 26087 --- [ocp-schedules-4,6138833039791592,080f2cf3a510e92f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:20:57.286 INFO 26087 --- [ocp-schedules-4,6138833039791592,080f2cf3a510e92f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:20:57.355 INFO 26087 --- [prometheus-minute-schedule-1,a3d0646293f65bc2,3b8ebfe68d4756c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118460, interval=60 2025-04-08 21:20:58.071 INFO 26087 --- [ocp-schedules-16,77d7890b5f3bab9e,bcd65abb66b2f1ed] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:20:58.072 INFO 26087 --- [ocp-schedules-16,77d7890b5f3bab9e,bcd65abb66b2f1ed] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:20:58.623 INFO 26087 --- [ocp-async-9,eed1a55b33f7484e,c01af8bb29689f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:20:58.623 WARN 26087 --- [ocp-async-3,37990f2c0cf2c81b,5451441003fd67cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:20:59.369 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:20:59.369 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:20:59.901 INFO 26087 --- [ocp-schedules-23,c7ace63d568ac6d4,c0285851b33de292] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:21:00.355 INFO 26087 --- [prometheus-minute-schedule-1,85c4ae5280131959,9b4d3895af33a0d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:00.623 WARN 26087 --- [ocp-async-2,0061ca1f7155beaa,7b311ceca37191c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:00.623 INFO 26087 --- [ocp-async-9,cd3c0318f0efd755,de9bcf25a9c133e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:02.369 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:02.370 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:02.623 WARN 26087 --- [ocp-async-9,da88cec71ecc3d32,d6e42dd6f318394d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:02.623 INFO 26087 --- [ocp-async-10,56760df9b80b1102,5e8afc53b9737420] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:03.355 INFO 26087 --- [prometheus-minute-schedule-1,7dc8e36c02d96311,1c8711d6da3723b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:04.623 WARN 26087 --- [ocp-async-10,b252b5cff0aefd3f,e306556952d9b98b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:04.623 INFO 26087 --- [ocp-async-3,fb194464337fd1ca,ee69192ca49e7d0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:05.370 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:05.370 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:06.355 INFO 26087 --- [prometheus-minute-schedule-1,7898bd7a8557b0ef,024db0a50d2101e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:06.355 INFO 26087 --- [prometheus-cache-monitor1,b0c9c6daa7f2af81,a1e08999c29feab1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1510,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:21:06.355 INFO 26087 --- [prometheus-cache-monitor1,b0c9c6daa7f2af81,a1e08999c29feab1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:21:06.554 INFO 26087 --- [ocp-async-1,a8897301dbfc9431,5216700f901c22e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:21:06.623 WARN 26087 --- [ocp-async-6,b82199c6ebf52245,404e90b40fbfe72e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:06.623 INFO 26087 --- [ocp-async-4,925ff3230d5287b1,662688a42b29b848] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:21:06.623377+08:00, lastProcessTime=2025-04-08T21:20:56.624163+08:00 2025-04-08 21:21:06.682 INFO 26087 --- [ocp-schedules-25,a8bdf9e50e13db6c,ea3d41e092974767] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:21:06.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:21:06.623377+08:00 2025-04-08 21:21:08.075 INFO 26087 --- [ocp-schedules-9,fe355bc516880bc2,9308ddf0a1dc9b72] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:21:08.076 INFO 26087 --- [ocp-schedules-9,fe355bc516880bc2,9308ddf0a1dc9b72] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:21:08.371 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:08.371 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:08.623 WARN 26087 --- [ocp-async-9,07b9744ee43b3073,441150f4bb6e6a84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:08.623 INFO 26087 --- [ocp-async-5,06d3a222fea27088,1d04d00243dc27de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:09.355 INFO 26087 --- [prometheus-minute-schedule-1,58e27eaaea61322e,6a2a045501f80297] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:09.920 INFO 26087 --- [ocp-schedules-16,6d6cea8fcb64943d,4b5502745e3e3eef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:21:10.001 INFO 26087 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:21:10.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:21:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:21:10.623 INFO 26087 --- [ocp-async-10,60ee40706d3522ab,b08270ac8e9d63e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:10.623 WARN 26087 --- [ocp-async-7,f6f7e65b6d6c3cfd,97a8eaa72dee36e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:11.371 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:11.372 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:12.355 INFO 26087 --- [prometheus-minute-schedule-1,039db3d635fa99c9,407af86c2588d755] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:12.435 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118400, nodeCount=92, affectRows=92 2025-04-08 21:21:12.435 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118400, nodeCount=92, validCount=92 2025-04-08 21:21:12.454 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118400, nodeCount=94, affectRows=94 2025-04-08 21:21:12.454 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118400, nodeCount=95, affectRows=95 2025-04-08 21:21:12.454 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118400, nodeCount=97, affectRows=97 2025-04-08 21:21:12.455 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118400, nodeCount=94, validCount=94 2025-04-08 21:21:12.455 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118400, nodeCount=94, validCount=94 2025-04-08 21:21:12.455 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118400, nodeCount=97, validCount=97 2025-04-08 21:21:12.624 WARN 26087 --- [ocp-async-1,3e98a285e351cf6f,2e15835aee428f97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:12.624 INFO 26087 --- [ocp-async-6,a732612be1de4845,56b2cb4ecbf3e28c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:14.372 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:14.372 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:14.624 WARN 26087 --- [ocp-async-8,f5d690ac94207731,5758dc2bd4bb0456] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:14.624 INFO 26087 --- [ocp-async-4,b4d868a3e746d677,140d7b2f6e1b54d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:15.355 INFO 26087 --- [prometheus-minute-schedule-1,d9e10a99b7af948e,39d2f012b1125489] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:16.517 INFO 26087 --- [ocp-schedules-13,e0d92a32d65ba0a9,06202992adc0ca2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:21:16.518 INFO 26087 --- [ocp-schedules-17,1c030fa0239007b1,1ed5b089799dbbb7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:21:16.520 INFO 26087 --- [ocp-schedules-20,6e2f210cbc061289,b71860c5d503a97a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:21:16.520 INFO 26087 --- [ocp-schedules-20,6e2f210cbc061289,b71860c5d503a97a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:21:16.521 INFO 26087 --- [ocp-schedules-6,21676c384865a39e,b533739da8c23ab5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:21:16.523 INFO 26087 --- [ocp-schedules-12,36ff75605d19bd05,00638cc4d7c6dce0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:21:16.523 INFO 26087 --- [ocp-schedules-12,36ff75605d19bd05,00638cc4d7c6dce0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:21:16.523 INFO 26087 --- [ocp-schedules-29,09a8fb914f965449,7cddcce667190b8b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:21:16.523 INFO 26087 --- [ocp-schedules-29,09a8fb914f965449,7cddcce667190b8b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:21:16.524 INFO 26087 --- [ocp-schedules-7,b5cc4c2f8fd8aef3,a60717bdec5699c5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:21:16.524 INFO 26087 --- [ocp-schedules-19,fe8c5d0da3fd0d83,6fb4d248088cb61e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:21:16.524 INFO 26087 --- [ocp-schedules-18,cd4ee198d1f4a6b4,4fad99db6069d2f4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:21:16.525 INFO 26087 --- [ocp-schedules-17,1c030fa0239007b1,1ed5b089799dbbb7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:21:16.528 INFO 26087 --- [ocp-schedules-13,e0d92a32d65ba0a9,06202992adc0ca2b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:21:16.554 INFO 26087 --- [ocp-async-7,aac1141cfe7dfe9a,e76efb113bd1721e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:21:16.557 INFO 26087 --- [ocp-schedules-24,125bdeb7a76093b7,a977296b1d3ec69f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:21:16.586 INFO 26087 --- [ocp-schedules-3,61f53ccb64aa888c,bba49c7618f62510] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:21:16.586 INFO 26087 --- [ocp-schedules-3,61f53ccb64aa888c,bba49c7618f62510] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:21:16.594 INFO 26087 --- [ocp-schedules-27,9a39a76e0f120e17,93977338efc2850f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:21:16.616 INFO 26087 --- [ocp-schedules-1,0e80c94cbb7c9450,eb6efef895b21586] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:21:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:21:16.622708+08:00, lastProcessTime=2025-04-08T21:21:06.623377+08:00 2025-04-08 21:21:16.624 INFO 26087 --- [ocp-async-10,bfff4b37777b30a1,fe5b1d9a97a69d35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:16.624 WARN 26087 --- [ocp-async-2,3d6446a6b27af22c,804f47946c0f9b84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:16.627 INFO 26087 --- [ocp-schedules-15,93f8280a57817346,59010257f321f580] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:21:16.627 INFO 26087 --- [ocp-schedules-15,93f8280a57817346,59010257f321f580] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:21:16.645 INFO 26087 --- [ocp-schedules-29,6d0ccd839fe1360b,80361a7dc776f1d7] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:21:16.645 INFO 26087 --- [ocp-schedules-29,6d0ccd839fe1360b,80361a7dc776f1d7] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:21:16.645 INFO 26087 --- [ocp-schedules-6,3d41c39b5e5740af,574232c67f7b92c1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:21:16.645+08:00, clusters=[] 2025-04-08 21:21:16.646 INFO 26087 --- [ocp-schedules-12,0e426b5ad3046395,665c67bb5b9a4a69] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:21:16.651 INFO 26087 --- [ocp-schedules-7,b5f0dbdba3ecbc68,0749c044c058ca2a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:21:16.651+08:00, clusters=[] 2025-04-08 21:21:16.669 INFO 26087 --- [ocp-schedules-18,97c84a60c3b29e6b,10bfe970232202a6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:21:16.669 INFO 26087 --- [ocp-schedules-18,97c84a60c3b29e6b,10bfe970232202a6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:21:16.670 INFO 26087 --- [ocp-schedules-18,97c84a60c3b29e6b,10bfe970232202a6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:21:16.671 INFO 26087 --- [ocp-schedules-18,97c84a60c3b29e6b,10bfe970232202a6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:21:16.676 INFO 26087 --- [ocp-schedules-17,5186fd58368d4d82,ed88d5dc4f4d1be9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:21:16.677 INFO 26087 --- [ocp-schedules-17,5186fd58368d4d82,ed88d5dc4f4d1be9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:21:16.677 INFO 26087 --- [ocp-schedules-17,5186fd58368d4d82,ed88d5dc4f4d1be9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:21:16.684 INFO 26087 --- [ocp-schedules-9,8762793b36a7971e,00f16d9533a55bc5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:21:16.713 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:21:16.622708+08:00 2025-04-08 21:21:16.777 INFO 26087 --- [ocp-schedules-4,25c09bca8df47824,548e7e4c77977c8b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:21:16.827 INFO 26087 --- [ocp-schedules-24,ad82e4c2a4c80cc5,4ad1621b8374ceb1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:21:16.831 INFO 26087 --- [ocp-schedules-14,d15b7d2585a5e86e,a8e2bc57d02f260f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:21:16.841 INFO 26087 --- [ocp-schedules-28,25fd1b0d23951b6d,376ea65ac68a0d25] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:21:16.846 INFO 26087 --- [ocp-schedules-22,97422b83abce92f4,a19583392233daef] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:21:16.846 INFO 26087 --- [ocp-schedules-22,97422b83abce92f4,a19583392233daef] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:21:16.847 INFO 26087 --- [ocp-schedules-22,97422b83abce92f4,a19583392233daef] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:21:16.847 INFO 26087 --- [ocp-schedules-22,97422b83abce92f4,a19583392233daef] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:21:16.847 INFO 26087 --- [ocp-schedules-22,97422b83abce92f4,a19583392233daef] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:21:16.847 INFO 26087 --- [ocp-schedules-22,97422b83abce92f4,a19583392233daef] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:21:17.287 INFO 26087 --- [ocp-schedules-16,e6b8a07685e34e33,f45bf17f70f611d4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:21:17.293 INFO 26087 --- [ocp-schedules-16,e6b8a07685e34e33,f45bf17f70f611d4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:21:17.293 INFO 26087 --- [ocp-schedules-16,e6b8a07685e34e33,f45bf17f70f611d4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:21:17.373 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:17.373 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:17.557 INFO 26087 --- [ocp-schedules-2,e082cc338368ac71,fa2d108a39ddb1ef] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:21:17.557 INFO 26087 --- [ocp-schedules-2,e082cc338368ac71,fa2d108a39ddb1ef] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:21:18.083 INFO 26087 --- [ocp-schedules-17,346c167f47059192,1a614f67b81de78d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:21:18.084 INFO 26087 --- [ocp-schedules-17,346c167f47059192,1a614f67b81de78d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:21:18.355 INFO 26087 --- [prometheus-minute-schedule-1,8ad439948bfc27db,3d4996a468a67617] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:18.624 INFO 26087 --- [ocp-async-6,66d35f5ca7d119d2,43b652b8d4866868] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:18.624 WARN 26087 --- [ocp-async-9,ab9c04d38390fdf3,913ee684b6a2df3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:19.936 INFO 26087 --- [ocp-schedules-15,d35e5633ff4bdd99,b6bf5cfc371e5c18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:21:20.373 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:20.374 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:21:20.624 WARN 26087 --- [ocp-async-4,1d0e4fd3f60107a5,f3b0851c4d21dbfc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:20.624 INFO 26087 --- [ocp-async-5,eb60c31e764130f3,773d1fe333eb300a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:21.355 INFO 26087 --- [prometheus-minute-schedule-1,407e0d689262f32f,b6dcb4042a91f0a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:22.624 WARN 26087 --- [ocp-async-3,9e731a3b16516737,b1739d99dc2777d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:22.624 INFO 26087 --- [ocp-async-1,1f66a1ebe1397127,fc857b04f2bf3dbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:23.374 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:23.374 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:24.082 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:21:24.082 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:21:24.085 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:21:24.085 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:21:24.085 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:21:24.085 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:21:24.097 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:21:24.097 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:21:24.097 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:21:24.097 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:21:24.122 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:21:24.127 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:21:24.127 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:21:24.127 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:21:24.127 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:21:24.127 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:21:24.132 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:21:24.355 INFO 26087 --- [prometheus-minute-schedule-1,78464040544528b6,7df969170e990b9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:24.437 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:21:24.624 INFO 26087 --- [ocp-async-10,48632533c6e9f023,f33860370ffc7607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:24.624 WARN 26087 --- [ocp-async-9,6e524a73dd6cd640,de4fc0f186a9ab97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:24.758 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.661600667S 2025-04-08 21:21:24.758 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:21:24.815 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:21:24.815 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:21:24.840 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:21:24.882 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:21:24.882 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:21:24.882 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:21:24.882 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:21:24.882 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:21:24.882 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:21:24.889 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:21:24.989 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] o.s.boot.SpringApplication : Started application in 0.903 seconds (JVM running for 1190.361) 2025-04-08 21:21:24.989 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.904120769S 2025-04-08 21:21:25.260 INFO 26087 --- [ocp-schedules-19,1ca4c346832013bf,7371e275957c9a69] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:21:26.374 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:26.375 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:26.554 INFO 26087 --- [ocp-async-6,21939c58205fc2b1,1da633d83f70f81c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:21:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:21:26.623824+08:00, lastProcessTime=2025-04-08T21:21:16.622708+08:00 2025-04-08 21:21:26.625 WARN 26087 --- [ocp-async-5,c2fcb25df115cffb,5e808b68a1bf9ba5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:26.625 INFO 26087 --- [ocp-async-7,6f94176f904db0b0,c0269d53bfd2a4ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:26.685 INFO 26087 --- [ocp-schedules-14,41e61be28375a3a9,f6dc3dd7968b71fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:21:26.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:21:26.623824+08:00 2025-04-08 21:21:27.355 INFO 26087 --- [prometheus-minute-schedule-1,8c2103908b1dcb98,d432cb185d8a1e48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:28.099 INFO 26087 --- [ocp-schedules-30,3fa75ff950aceabc,7cac7c5cdeec1743] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:21:28.101 INFO 26087 --- [ocp-schedules-30,3fa75ff950aceabc,7cac7c5cdeec1743] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:21:28.625 WARN 26087 --- [ocp-async-1,5153a8dab8d7ac85,32919e4ef85ce2a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:28.625 INFO 26087 --- [ocp-async-7,c4803dd2788ab176,3fd7a14c33487183] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:29.375 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:29.376 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:29.958 INFO 26087 --- [ocp-schedules-13,b03fe2f586e26b5f,d7ba23d1c19ea05d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:21:30.355 INFO 26087 --- [prometheus-minute-schedule-1,d1b3a5288a3ea395,cfe28d3524ef3439] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:30.625 WARN 26087 --- [ocp-async-8,7345fdea760878ea,516f86ffdc0574a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:30.625 INFO 26087 --- [ocp-async-10,ca85837d9beed87f,f53b0ff0f1874446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:32.376 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:32.376 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:32.625 WARN 26087 --- [ocp-async-5,5c80ca7328dc9962,8b4cdfbd8dcdfa60] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:32.625 INFO 26087 --- [ocp-async-10,148bdb9cd927451b,ede85014e241aafe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:33.355 INFO 26087 --- [prometheus-minute-schedule-1,1314dc2094490581,ebddbe100db38f8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:34.625 WARN 26087 --- [ocp-async-1,af020da15205eb7b,c2351b2a93f7553d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:34.625 INFO 26087 --- [ocp-async-10,01de3240593b9795,5b9973815d7abe3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:34.995 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:21:34.995 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:21:34.996 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:21:34.996 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:21:34.996 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:21:34.997 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:21:35.376 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:35.377 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:36.355 INFO 26087 --- [prometheus-minute-schedule-1,3a04efe6fdd9c9b0,3ac499c922dd5959] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:36.555 INFO 26087 --- [ocp-async-8,4a142a36071c11b2,411b3aef743628eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:21:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:21:36.622977+08:00, lastProcessTime=2025-04-08T21:21:26.623824+08:00 2025-04-08 21:21:36.625 WARN 26087 --- [ocp-async-7,fbed349f8d9bc459,adfc52bee0d10ca5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:36.625 INFO 26087 --- [ocp-async-10,f6a54708877c70f0,cca8535c7957aced] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:36.686 INFO 26087 --- [ocp-schedules-9,e703e66938348713,8f3bc8dd8931f3ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:21:36.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:21:36.622977+08:00 2025-04-08 21:21:37.295 INFO 26087 --- [ocp-schedules-15,d1e4b666aeb04f8a,dd9903b29128826e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:21:37.303 INFO 26087 --- [ocp-schedules-15,d1e4b666aeb04f8a,dd9903b29128826e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:21:37.303 INFO 26087 --- [ocp-schedules-15,d1e4b666aeb04f8a,dd9903b29128826e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:21:38.107 INFO 26087 --- [ocp-schedules-6,a9f1a7f60d5056e5,396100d79dd75758] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:21:38.108 INFO 26087 --- [ocp-schedules-6,a9f1a7f60d5056e5,396100d79dd75758] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:21:38.377 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:38.378 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:38.625 WARN 26087 --- [ocp-async-6,7579aea45d4eba64,4d8beaa5e3a31b04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:38.625 INFO 26087 --- [ocp-async-3,04fbae9ab3fcfece,edc78dc397349bc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:39.355 INFO 26087 --- [prometheus-minute-schedule-1,f2ce8469653a248d,97bd3bcabc677f65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:39.990 INFO 26087 --- [ocp-schedules-17,3b18051a0155e6b6,e89d2025545c4360] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:21:40.001 INFO 26087 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:21:40.002 INFO 26087 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:21:40.626 WARN 26087 --- [ocp-async-9,6351804da7f42016,2afc84382fb5825e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:40.626 INFO 26087 --- [ocp-async-2,3fbdcbe6c85068ea,12b9b0c2823ce45b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:41.378 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:41.378 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:42.355 INFO 26087 --- [prometheus-minute-schedule-1,26ed05ed4f5ab048,97840fb274704e57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:42.626 WARN 26087 --- [ocp-async-7,9cf7049e3a17e643,41c529ba8857fdc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:42.626 INFO 26087 --- [ocp-async-2,56467ea892be272f,00dd2a19e30c2b7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:44.378 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:44.379 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:44.626 WARN 26087 --- [ocp-async-6,793644e8fb7af01e,e3073a9ef965d8d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:44.626 INFO 26087 --- [ocp-async-2,412951fd96c440e7,e40817e96b6dc918] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:45.355 INFO 26087 --- [prometheus-minute-schedule-1,3a7cc86881d243ad,1412d2d6ce2a67e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:46.521 INFO 26087 --- [ocp-schedules-12,60c4addefe64e7bc,5bc5469728a65ce2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:21:46.521 INFO 26087 --- [ocp-schedules-19,f23c08369b00a42e,c4f831d1ec40dd4b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:21:46.521 INFO 26087 --- [ocp-schedules-19,f23c08369b00a42e,c4f831d1ec40dd4b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:21:46.523 INFO 26087 --- [ocp-schedules-11,b3151280a7d8c19a,eda27fe5f8053f6f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:21:46.523 INFO 26087 --- [ocp-schedules-16,c5847afc20fee36d,875c0e52bde2f33a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:21:46.523 INFO 26087 --- [ocp-schedules-11,b3151280a7d8c19a,eda27fe5f8053f6f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:21:46.523 INFO 26087 --- [ocp-schedules-16,c5847afc20fee36d,875c0e52bde2f33a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:21:46.524 INFO 26087 --- [ocp-schedules-4,958bb6b7c96280bc,1d6c64b4ddc560ce] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:21:46.524 INFO 26087 --- [ocp-schedules-26,37ed669461e13e77,0c8a80fa8fcdae53] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:21:46.524 INFO 26087 --- [ocp-schedules-10,6931a62348895f35,79686a831c5f6c4f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:21:46.528 INFO 26087 --- [ocp-schedules-25,8895773d7472745c,77484e16f4d0ff77] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:21:46.555 INFO 26087 --- [ocp-async-2,1e758b28290b7b4f,83310ffb5589803e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:21:46.617 INFO 26087 --- [ocp-schedules-8,bec7337789dca807,a46c1564bc61b20a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:21:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:21:46.623496+08:00, lastProcessTime=2025-04-08T21:21:36.622977+08:00 2025-04-08 21:21:46.626 WARN 26087 --- [ocp-async-8,d6d01d89adad0516,9933daed221efcae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:46.626 INFO 26087 --- [ocp-async-9,461863effe28070d,77db9b4da9c1c4bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:46.687 INFO 26087 --- [ocp-schedules-7,d441ed6dc48470b0,3661625a65206212] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:21:46.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:21:46.623496+08:00 2025-04-08 21:21:47.379 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:47.380 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:48.115 INFO 26087 --- [ocp-schedules-11,6ff7304ea4643850,81d106889120c704] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:21:48.116 INFO 26087 --- [ocp-schedules-11,6ff7304ea4643850,81d106889120c704] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:21:48.355 INFO 26087 --- [prometheus-minute-schedule-1,cddb972e967e6e24,015af4b22e210001] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:48.626 INFO 26087 --- [ocp-async-7,b20072d4c96d9c44,6e406838e3284a15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:48.626 WARN 26087 --- [ocp-async-4,3ff876a948473a14,f0b6837d3100e89a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:50.011 INFO 26087 --- [ocp-schedules-27,2ae5cd4fd216802e,449fb7b83a32fdcd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:21:50.380 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:50.381 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:50.626 WARN 26087 --- [ocp-async-6,4d93ad5c94eca8db,f699770debe45bbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:50.626 INFO 26087 --- [ocp-async-10,7312fb96f453277c,0685907c8383c35c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:51.355 INFO 26087 --- [prometheus-minute-schedule-1,34e9c3b24c33d496,bcaa8b24d21b0eca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:52.627 WARN 26087 --- [ocp-async-3,ce0f95e126b4e027,178b6c01eb033fbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:52.627 INFO 26087 --- [ocp-async-2,05507ae2e06cb604,fefbbf71d4e1016c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:53.381 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:53.381 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:54.355 INFO 26087 --- [prometheus-minute-schedule-1,34dc48a435993e39,21e3ef5c7eb070fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:54.627 WARN 26087 --- [ocp-async-4,1a6a471a3b2e760d,fdc1899dff4f4005] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:54.627 INFO 26087 --- [ocp-async-5,080865bc32485136,c876f529ccdd1525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:56.381 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:56.382 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:21:56.555 INFO 26087 --- [ocp-async-9,2f3542372940891e,bca51bc724dacee9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:21:56.571 INFO 26087 --- [ocp-schedules-5,4e9ddf19156ed0c1,b12604ff494f3368] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:21:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:21:56.623112+08:00, lastProcessTime=2025-04-08T21:21:46.623496+08:00 2025-04-08 21:21:56.627 WARN 26087 --- [ocp-async-1,4c05749013b792bd,9ffdc0b375773ef9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:56.627 INFO 26087 --- [ocp-async-10,941a2a23f3e901ce,54c6742c23a7a7ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:56.689 INFO 26087 --- [ocp-schedules-26,8f221441707cf5bc,f73e7c831e4107c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:21:56.703 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:21:56.623112+08:00 2025-04-08 21:21:57.305 INFO 26087 --- [ocp-schedules-24,3cc18d4858f4846f,29c052cfa448a3dc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:21:57.320 INFO 26087 --- [ocp-schedules-24,3cc18d4858f4846f,29c052cfa448a3dc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:21:57.320 INFO 26087 --- [ocp-schedules-24,3cc18d4858f4846f,29c052cfa448a3dc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:21:57.355 INFO 26087 --- [prometheus-minute-schedule-1,347ebd574e276c33,c8df56db5a5e99f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118520, interval=60 2025-04-08 21:21:58.121 INFO 26087 --- [ocp-schedules-27,7675a31dd61439f0,89a60c63a61b2f21] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:21:58.122 INFO 26087 --- [ocp-schedules-27,7675a31dd61439f0,89a60c63a61b2f21] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:21:58.627 WARN 26087 --- [ocp-async-2,cf66b4caca41281f,d158d38ff2f51c72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:21:58.627 INFO 26087 --- [ocp-async-8,38aa4128fd837f07,ac6eeb097aaf810c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:21:59.382 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:21:59.382 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:00.029 INFO 26087 --- [ocp-schedules-25,b6c701c88ad327e0,77e1d7a7bdfda7be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:22:00.355 INFO 26087 --- [prometheus-minute-schedule-1,19294ced7620dd3f,7db513c618a31676] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:00.627 WARN 26087 --- [ocp-async-7,2b3a379065b1945f,bd4d69a3cb06e34f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:00.627 INFO 26087 --- [ocp-async-8,e39022dacc533f95,9637de02fe977e95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:02.382 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:02.383 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:02.627 WARN 26087 --- [ocp-async-1,7179f93bf33201c4,e85fec71e41e4400] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:02.627 INFO 26087 --- [ocp-async-9,71d32c77a4ebdb5e,8589ba0aeb6cce4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:03.355 INFO 26087 --- [prometheus-minute-schedule-1,cacbbb4f26a9a20d,fd3dfa2130ac2f96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:04.627 WARN 26087 --- [ocp-async-9,1cfb428277f90255,e06ada1f8cd763b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:04.627 INFO 26087 --- [ocp-async-2,6b29f6c12a6ec8e8,c202215a65ab409b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:05.383 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:05.384 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:06.355 INFO 26087 --- [prometheus-minute-schedule-1,d018996e8362a950,4e308a3a2555a47c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:06.355 INFO 26087 --- [prometheus-cache-monitor1,0dfdb6a9479f2e09,412bd6f414df8b11] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:22:06.355 INFO 26087 --- [prometheus-cache-monitor1,0dfdb6a9479f2e09,412bd6f414df8b11] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:22:06.482 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:18:06.44, end=2025-04-08T21:20:06.456} 2025-04-08 21:22:06.487 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:22:06.487 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:22:06.555 INFO 26087 --- [ocp-async-2,2d5f5ea4b184db49,7843417b811bf12d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:22:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:22:06.623148+08:00, lastProcessTime=2025-04-08T21:21:56.623112+08:00 2025-04-08 21:22:06.628 WARN 26087 --- [ocp-async-7,107cfbf898619349,2dfc725785893c5c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:06.628 INFO 26087 --- [ocp-async-5,42885deab2a9452b,dc5f78b97379ce16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:06.690 INFO 26087 --- [ocp-schedules-26,f7cc09a986ae7189,58f8cf797f990b43] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:22:06.737 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:22:06.623148+08:00 2025-04-08 21:22:08.127 INFO 26087 --- [ocp-schedules-17,0857fd50b6802d94,0b889adb32664862] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:22:08.129 INFO 26087 --- [ocp-schedules-17,0857fd50b6802d94,0b889adb32664862] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:22:08.384 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:08.384 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:08.628 WARN 26087 --- [ocp-async-6,98a37ece60b24b90,c39d774867c456d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:08.628 INFO 26087 --- [ocp-async-10,a16e55cad2c94e0a,fee901e1c3ce9711] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:09.355 INFO 26087 --- [prometheus-minute-schedule-1,572d230cfbf37d33,43c81acf293df6b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:10.002 INFO 26087 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:22:10.002 INFO 26087 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:22:10.051 INFO 26087 --- [ocp-schedules-28,ea5fe721cfbe9156,c26c2b28ac003346] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:22:10.605 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:22:10.628 WARN 26087 --- [ocp-async-9,2d24a12119c8e255,b42b405aaaa5192f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:10.628 INFO 26087 --- [ocp-async-2,03d8e8e7fdf0c0c6,0689d1191c623a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:11.384 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:11.385 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:12.355 INFO 26087 --- [prometheus-minute-schedule-1,e94650c828ecd8fc,c7d74f198792ae63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:12.425 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=35, affectRows=35 2025-04-08 21:22:12.425 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=35, validCount=35 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=37, affectRows=37 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=48, affectRows=48 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=35, affectRows=35 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=53, affectRows=53 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=65, affectRows=65 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=37, validCount=37 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=65, affectRows=65 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=35, validCount=35 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=48, validCount=48 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=53, validCount=53 2025-04-08 21:22:12.426 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118460, nodeCount=40, affectRows=40 2025-04-08 21:22:12.427 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=64, validCount=64 2025-04-08 21:22:12.427 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=65, validCount=65 2025-04-08 21:22:12.427 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118460, nodeCount=40, validCount=40 2025-04-08 21:22:12.628 WARN 26087 --- [ocp-async-7,0a17e52b3686ec94,81a6f1064f11a951] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:12.628 INFO 26087 --- [ocp-async-3,0ded11949c7dad47,6d4085e48cda558b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:14.385 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:14.386 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:14.628 WARN 26087 --- [ocp-async-6,85b4715e0f9901fb,83fc8ab18a481dd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:14.628 INFO 26087 --- [ocp-async-5,1496b159f231e86d,3e47ba71de9bb0b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:15.355 INFO 26087 --- [prometheus-minute-schedule-1,2caf197ffc911d7d,c1d6d59022460875] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:16.517 INFO 26087 --- [ocp-schedules-13,7b6c8330d0cfd219,d07769dc0adc4de6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:22:16.518 INFO 26087 --- [ocp-schedules-27,be901af88ad7e2c1,2b0fba7b12e105ff] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:22:16.521 INFO 26087 --- [ocp-schedules-2,4033fb9ef9da5a5c,69ee84abb0b41c7d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:22:16.521 INFO 26087 --- [ocp-schedules-13,7b6c8330d0cfd219,d07769dc0adc4de6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:22:16.521 INFO 26087 --- [ocp-schedules-29,471a46cbb2694b7c,28cb5fd30f25d19c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:22:16.521 INFO 26087 --- [ocp-schedules-29,471a46cbb2694b7c,28cb5fd30f25d19c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:22:16.523 INFO 26087 --- [ocp-schedules-30,af9b77e51ca0a772,0cafd4be0445b00d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:22:16.523 INFO 26087 --- [ocp-schedules-30,af9b77e51ca0a772,0cafd4be0445b00d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:22:16.524 INFO 26087 --- [ocp-schedules-22,45660379f787c6b2,bd845945956a8732] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:22:16.524 INFO 26087 --- [ocp-schedules-22,45660379f787c6b2,bd845945956a8732] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:22:16.524 INFO 26087 --- [ocp-schedules-22,aca0191d3b9dbdfe,bba050bf23cc0ce8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:22:16.524 INFO 26087 --- [ocp-schedules-26,447a3ea8b67e1aaa,b4e8b5bed9031c2e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:22:16.524 INFO 26087 --- [ocp-schedules-16,1c294a7850258fd0,f05b7098dbdcc709] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:22:16.525 INFO 26087 --- [ocp-schedules-27,be901af88ad7e2c1,2b0fba7b12e105ff] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:22:16.527 INFO 26087 --- [ocp-schedules-17,792ce9f96845e450,62095c0ad591c5e6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:22:16.555 INFO 26087 --- [ocp-async-9,394788ad50598d39,09c95f43f2ac6113] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:22:16.561 INFO 26087 --- [ocp-schedules-21,5cfda1d343025a76,60787b4f56662630] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:22:16.561 INFO 26087 --- [ocp-schedules-21,5cfda1d343025a76,60787b4f56662630] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:22:16.595 INFO 26087 --- [ocp-schedules-3,b343cabb840f6f53,a192a0ccbbc014ef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:22:16.618 INFO 26087 --- [ocp-schedules-4,be9da419a79b1712,6f9d031f0e9111cc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:22:16.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:22:16.624510+08:00, lastProcessTime=2025-04-08T21:22:06.623148+08:00 2025-04-08 21:22:16.628 WARN 26087 --- [ocp-async-7,716e8275edebfe11,757c8356574a06ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:16.628 INFO 26087 --- [ocp-async-8,ef3f0a63c3d21aa5,19ff5b334282ad8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:16.629 INFO 26087 --- [ocp-schedules-11,d42c7ffb32b515d8,237c91ac4e0c705f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:22:16.629 INFO 26087 --- [ocp-schedules-11,d42c7ffb32b515d8,237c91ac4e0c705f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:22:16.646 INFO 26087 --- [ocp-schedules-2,f190113a54c95bec,7550ec0574158447] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:22:16.646 INFO 26087 --- [ocp-schedules-2,f190113a54c95bec,7550ec0574158447] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:22:16.646 INFO 26087 --- [ocp-schedules-7,287dd88d604bf9da,6c8c5fe467661f29] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:22:16.646+08:00, clusters=[] 2025-04-08 21:22:16.646 INFO 26087 --- [ocp-schedules-30,5d28f92df65a07ec,f79ccac9189d65c4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:22:16.652 INFO 26087 --- [ocp-schedules-26,ee2667a2643204f6,2dccac8dd0f35fa6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:22:16.652+08:00, clusters=[] 2025-04-08 21:22:16.671 INFO 26087 --- [ocp-schedules-27,8cc0062be5ab1381,be30a503e07aa83b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:22:16.671 INFO 26087 --- [ocp-schedules-27,8cc0062be5ab1381,be30a503e07aa83b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:22:16.672 INFO 26087 --- [ocp-schedules-27,8cc0062be5ab1381,be30a503e07aa83b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:22:16.672 INFO 26087 --- [ocp-schedules-27,8cc0062be5ab1381,be30a503e07aa83b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:22:16.677 INFO 26087 --- [ocp-schedules-22,5640653db81198cb,18d0b8d7554dd795] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:22:16.678 INFO 26087 --- [ocp-schedules-22,5640653db81198cb,18d0b8d7554dd795] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:22:16.678 INFO 26087 --- [ocp-schedules-22,5640653db81198cb,18d0b8d7554dd795] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:22:16.691 INFO 26087 --- [ocp-schedules-17,773fa0159e17488d,66ac5e035871752b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:22:16.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:22:16.624510+08:00 2025-04-08 21:22:16.782 INFO 26087 --- [ocp-schedules-28,7c37abe9b430478d,26d1acb7ef927bae] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:22:16.828 INFO 26087 --- [ocp-schedules-19,c05213bc2f2082c9,4fcfc8d25178d5d0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:22:16.842 INFO 26087 --- [ocp-schedules-8,a3e085bf592314f4,6ef1e8225df07e66] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:22:16.848 INFO 26087 --- [ocp-schedules-21,5cc4da1053068bcf,cd7e3f4c5e74e025] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:22:16.848 INFO 26087 --- [ocp-schedules-21,5cc4da1053068bcf,cd7e3f4c5e74e025] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:22:16.851 INFO 26087 --- [ocp-schedules-21,5cc4da1053068bcf,cd7e3f4c5e74e025] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:22:16.851 INFO 26087 --- [ocp-schedules-21,5cc4da1053068bcf,cd7e3f4c5e74e025] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:22:16.853 INFO 26087 --- [ocp-schedules-21,5cc4da1053068bcf,cd7e3f4c5e74e025] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:22:16.853 INFO 26087 --- [ocp-schedules-21,5cc4da1053068bcf,cd7e3f4c5e74e025] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:22:17.322 INFO 26087 --- [ocp-schedules-12,ccf9d14ff5bfeb53,cae54f8333fc4427] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:22:17.334 INFO 26087 --- [ocp-schedules-12,ccf9d14ff5bfeb53,cae54f8333fc4427] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:22:17.334 INFO 26087 --- [ocp-schedules-12,ccf9d14ff5bfeb53,cae54f8333fc4427] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:22:17.386 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:17.387 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:17.560 INFO 26087 --- [ocp-schedules-29,eb185a56a84762b3,4b3ed25c533cb93b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:22:17.560 INFO 26087 --- [ocp-schedules-29,eb185a56a84762b3,4b3ed25c533cb93b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:22:18.136 INFO 26087 --- [ocp-schedules-17,4ecb1e02bb347f46,18d7e35d48c7107c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:22:18.137 INFO 26087 --- [ocp-schedules-17,4ecb1e02bb347f46,18d7e35d48c7107c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:22:18.355 INFO 26087 --- [prometheus-minute-schedule-1,521c13a3204f4358,b113163405a4f1da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:18.628 WARN 26087 --- [ocp-async-1,178eb77422a28944,cffbcac27fb13534] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:18.628 INFO 26087 --- [ocp-async-6,d9b9bc19a2065add,c0afd0fa5860f360] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:20.068 INFO 26087 --- [ocp-schedules-10,9e8ad34173d5868d,ce4b55ae10cbc4e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:22:20.387 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:20.387 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:22:20.628 WARN 26087 --- [ocp-async-10,0308893567bcea77,9e07095e0d2d9f0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:20.629 INFO 26087 --- [ocp-async-5,e7f467039dd7fcd9,e29a1fbb20a4069f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:21.355 INFO 26087 --- [prometheus-minute-schedule-1,b86cbab225ccbaf2,318bcad19ffcab49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:22.629 WARN 26087 --- [ocp-async-4,92f4dfec872cd38d,0ae76b3d8d497689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:22.629 INFO 26087 --- [ocp-async-7,70f151523cd580e9,c6de527a108943fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:23.387 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:23.388 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:24.355 INFO 26087 --- [prometheus-minute-schedule-1,ee46fefcf514b769,894a996efb5c807b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:24.629 WARN 26087 --- [ocp-async-8,525c2ed33deefc7a,b250bad70a0cd11b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:24.629 INFO 26087 --- [ocp-async-3,bdfb8fdf0d89e479,51f93633ece0ce50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:26.388 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:26.389 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:26.555 INFO 26087 --- [ocp-async-6,e6dd93aa08046197,ca6600af8c51c8d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:22:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:22:26.623436+08:00, lastProcessTime=2025-04-08T21:22:16.624510+08:00 2025-04-08 21:22:26.629 WARN 26087 --- [ocp-async-4,f00bda8bd1c2f021,76efe1ed322de727] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:26.629 INFO 26087 --- [ocp-async-5,e55e4b6e5a8878d0,d99752e31b74fad9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:26.692 INFO 26087 --- [ocp-schedules-25,bdfd75e7c003981c,2a607e6f9125aec2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:22:26.757 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:22:26.623436+08:00 2025-04-08 21:22:27.355 INFO 26087 --- [prometheus-minute-schedule-1,afca759f52cd6f3b,611e57030772a357] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:28.140 INFO 26087 --- [ocp-schedules-23,be4190add3f66d33,33ca65661023e1ca] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:22:28.141 INFO 26087 --- [ocp-schedules-23,be4190add3f66d33,33ca65661023e1ca] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:22:28.629 WARN 26087 --- [ocp-async-8,78986a72126c9131,97b1253f81827d32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:28.629 INFO 26087 --- [ocp-async-7,c4654e24203191b9,ee2dee17a9ede613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:29.389 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:29.390 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:30.094 INFO 26087 --- [ocp-schedules-16,6a0ffc132fee5794,884c089218de62aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:22:30.355 INFO 26087 --- [prometheus-minute-schedule-1,7b397b09fbd8ee3b,c7a623c8f733600a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:30.629 INFO 26087 --- [ocp-async-1,7aa77d7d2f06badb,fba2c2b424296282] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:30.629 WARN 26087 --- [ocp-async-6,6beb8cdcfb7183b4,13db2b2c79b9c895] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:32.390 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:32.390 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:32.629 WARN 26087 --- [ocp-async-4,acaaf1d04b313eb5,c4ec5fe8ed2f83d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:32.629 INFO 26087 --- [ocp-async-1,6609e120ea837a66,279021c956d5a77f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:33.355 INFO 26087 --- [prometheus-minute-schedule-1,36b6dcc65aa3192e,b29a6b49f18cc172] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:34.629 WARN 26087 --- [ocp-async-1,f1493d345d38d65f,7ddd1d1be434f948] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:34.629 INFO 26087 --- [ocp-async-5,63a402e2f561b9e8,b2e194406eec20d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:35.390 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:35.391 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:36.355 INFO 26087 --- [prometheus-minute-schedule-1,4a17e04f513dada9,9afc1d13640a6e35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:36.556 INFO 26087 --- [ocp-async-5,d4f824da59a1b57b,753715bc7f527637] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:22:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:22:36.622881+08:00, lastProcessTime=2025-04-08T21:22:26.623436+08:00 2025-04-08 21:22:36.630 WARN 26087 --- [ocp-async-7,cfcf374568c17411,3045311e9bf07dce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:36.630 INFO 26087 --- [ocp-async-6,3755da09451a79f1,3bfa719e733136cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:36.693 INFO 26087 --- [ocp-schedules-18,023e2597d33d960b,39beeb857bbf54c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:22:36.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:22:36.622881+08:00 2025-04-08 21:22:37.335 INFO 26087 --- [ocp-schedules-17,18f9cbfb80e176c4,1067c55656b25d03] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:22:37.339 INFO 26087 --- [ocp-schedules-17,18f9cbfb80e176c4,1067c55656b25d03] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:22:37.339 INFO 26087 --- [ocp-schedules-17,18f9cbfb80e176c4,1067c55656b25d03] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:22:38.146 INFO 26087 --- [ocp-schedules-10,2dd84c6b3ea3720b,c7f15d497ef271a9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:22:38.148 INFO 26087 --- [ocp-schedules-10,2dd84c6b3ea3720b,c7f15d497ef271a9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:22:38.391 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:38.392 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:38.630 WARN 26087 --- [ocp-async-9,9b6c04fd4a4c92e1,5a13a4b3b160834e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:38.630 INFO 26087 --- [ocp-async-10,741d952c47fc2357,9d858cf2846eff3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:39.355 INFO 26087 --- [prometheus-minute-schedule-1,5b2e948f6f3ac6ee,c948377c494a5d60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:40.001 INFO 26087 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:22:40.001 INFO 26087 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:22:40.112 INFO 26087 --- [ocp-schedules-17,4b656c4a25d39aff,db0a5561c61a072f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:22:40.630 WARN 26087 --- [ocp-async-2,d51ecc75c241eb92,c00a38bc1b4a3d6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:40.630 INFO 26087 --- [ocp-async-8,3ea72c8392b67172,be94857088673aaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:41.392 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:41.392 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:42.355 INFO 26087 --- [prometheus-minute-schedule-1,93aa223c7ba81da4,42e7eae6de865964] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:42.630 INFO 26087 --- [ocp-async-8,c5c976792cd5f63b,e884561babac71c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:42.630 WARN 26087 --- [ocp-async-7,413c911450f08f73,2d89f91f376b4edd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:44.393 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:44.393 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:44.630 WARN 26087 --- [ocp-async-6,b12cc5cc266c27d0,3081c2386bb5e6dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:44.630 INFO 26087 --- [ocp-async-8,e74b0d67d10551c3,dfa7ac7850f18c20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:45.355 INFO 26087 --- [prometheus-minute-schedule-1,a548d2859f5dbaa3,856f85798d35bcbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:46.521 INFO 26087 --- [ocp-schedules-15,8d45b8e25d901727,0ff6acb84129978f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:22:46.521 INFO 26087 --- [ocp-schedules-17,e8ca7e5302f2febf,cdb236a17a70a5d7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:22:46.521 INFO 26087 --- [ocp-schedules-15,8d45b8e25d901727,0ff6acb84129978f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:22:46.523 INFO 26087 --- [ocp-schedules-12,d907c846af7a466b,f1849c45b8aa9a48] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:22:46.523 INFO 26087 --- [ocp-schedules-14,173f9528f7fd6e92,ef609d804bf9ab03] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:22:46.523 INFO 26087 --- [ocp-schedules-12,d907c846af7a466b,f1849c45b8aa9a48] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:22:46.523 INFO 26087 --- [ocp-schedules-14,173f9528f7fd6e92,ef609d804bf9ab03] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:22:46.524 INFO 26087 --- [ocp-schedules-29,3b0e453e611bfe35,028ac92781ea44fd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:22:46.524 INFO 26087 --- [ocp-schedules-18,83204ccfb2530012,1b43ef0926a64796] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:22:46.524 INFO 26087 --- [ocp-schedules-9,d4f2e52f1a332281,af3a8c2a2eed4ff0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:22:46.525 INFO 26087 --- [ocp-schedules-15,6da4cc505ef87cff,3a80e7939c3c3655] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:22:46.556 INFO 26087 --- [ocp-async-8,3a59d4af0baebc08,42d23d6c88f095f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:22:46.619 INFO 26087 --- [ocp-schedules-1,faea8ed362ff8b14,837af8ee3071eb28] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:22:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:22:46.622639+08:00, lastProcessTime=2025-04-08T21:22:36.622881+08:00 2025-04-08 21:22:46.630 WARN 26087 --- [ocp-async-10,25577cfb3700037e,e5be3970066847df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:46.630 INFO 26087 --- [ocp-async-2,f9510efa43a9aebc,55c355c525556a62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:46.694 INFO 26087 --- [ocp-schedules-19,2c0f6edaec363d22,e37fd4d2f8e77d29] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:22:46.729 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:22:46.622639+08:00 2025-04-08 21:22:47.393 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:47.394 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:48.154 INFO 26087 --- [ocp-schedules-10,a4f7590f8db35d34,c4f4890862892f37] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:22:48.156 INFO 26087 --- [ocp-schedules-10,a4f7590f8db35d34,c4f4890862892f37] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:22:48.355 INFO 26087 --- [prometheus-minute-schedule-1,fb58f8b44ee051fb,d253f39243b1b21e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:48.631 WARN 26087 --- [ocp-async-4,482482e0b0d2cd1e,6f01756fc99bae92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:48.631 INFO 26087 --- [ocp-async-7,9760d3bf8f9498dc,07a548b1b986faf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:50.147 INFO 26087 --- [ocp-schedules-21,218abf3192d16a51,18bf310f959c323e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:22:50.394 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:50.394 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:50.631 WARN 26087 --- [ocp-async-7,d53a7bf3daa81036,7e7a91b8f1cac900] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:50.631 INFO 26087 --- [ocp-async-9,29108f06bf5c1f7d,796071cef4124fca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:51.355 INFO 26087 --- [prometheus-minute-schedule-1,b37dcb18845c5c88,b74f1a74fbc77909] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:52.631 WARN 26087 --- [ocp-async-10,7b5b2d7b5db1ed88,e07061f0698c52e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:52.631 INFO 26087 --- [ocp-async-5,6eeaf671cb005b84,d10f0dfd47e15abb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:53.394 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:53.395 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:54.355 INFO 26087 --- [prometheus-minute-schedule-1,16fe56a5bb234404,c531bce11e1083a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:54.631 WARN 26087 --- [ocp-async-5,a501f99ff3d12513,6e6b5dbd7015d212] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:54.631 INFO 26087 --- [ocp-async-4,89d52674344857b4,b334c4c1a10c182a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:56.395 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:56.396 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:22:56.556 INFO 26087 --- [ocp-async-3,f3a15b488184d7c8,8e985aa98c94cf6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:22:56.572 INFO 26087 --- [ocp-schedules-9,3707f6d057076e47,09276da593c1d383] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:22:56.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:22:56.624884+08:00, lastProcessTime=2025-04-08T21:22:46.622639+08:00 2025-04-08 21:22:56.631 INFO 26087 --- [ocp-async-7,613294eee9c880c7,7283c265aa05dd3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:56.631 WARN 26087 --- [ocp-async-1,841c4697bfe0f397,7500ce8de42e8c04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:56.695 INFO 26087 --- [ocp-schedules-8,9288731e2d54afdf,4b7681275da8d99e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:22:56.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:22:56.624884+08:00 2025-04-08 21:22:57.341 INFO 26087 --- [ocp-schedules-28,ac96760c7d9684ac,6bd7437206c8da43] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:22:57.347 INFO 26087 --- [ocp-schedules-28,ac96760c7d9684ac,6bd7437206c8da43] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:22:57.348 INFO 26087 --- [ocp-schedules-28,ac96760c7d9684ac,6bd7437206c8da43] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:22:57.355 INFO 26087 --- [prometheus-minute-schedule-1,fcb2f9fd10d0e32d,e3045cd30d5bf234] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118580, interval=60 2025-04-08 21:22:58.158 INFO 26087 --- [ocp-schedules-7,eb4f99a9ec31c4dd,ce8e7b0156f1dbc8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:22:58.160 INFO 26087 --- [ocp-schedules-7,eb4f99a9ec31c4dd,ce8e7b0156f1dbc8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:22:58.631 WARN 26087 --- [ocp-async-8,c3b6b7a91994c05c,6701b7413b130615] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:22:58.631 INFO 26087 --- [ocp-async-7,6b3e8b407b695060,7aa9a182b4b2e8b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:22:59.396 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:22:59.396 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:00.164 INFO 26087 --- [ocp-schedules-12,3d6aed9ea5d22e4b,5fef04871cf66159] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:23:00.355 INFO 26087 --- [prometheus-minute-schedule-1,d1bdaa008120d075,6e6a263786c7bc8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:00.631 WARN 26087 --- [ocp-async-6,5be32669ef9e2240,51d48428203ff13a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:00.631 INFO 26087 --- [ocp-async-5,4547cf170bf945fb,e6ef4fb64e3a600d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:02.396 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:02.397 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:02.632 WARN 26087 --- [ocp-async-9,4e6a4cf1b024a829,db481517fa087f3e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:02.632 INFO 26087 --- [ocp-async-3,2b07768940d8eb22,563c8f022f3d5a60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:03.355 INFO 26087 --- [prometheus-minute-schedule-1,670f78e21e677fed,8a0022c3e833199b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:04.632 WARN 26087 --- [ocp-async-8,cc4baac728277ef0,4192b557ba6e3390] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:04.632 INFO 26087 --- [ocp-async-3,d9a75fce5313c085,f9a95c40c1126d7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:05.397 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:05.398 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:06.355 INFO 26087 --- [prometheus-minute-schedule-1,a53e0fedce6e184c,d0dc036384581b5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:06.355 INFO 26087 --- [prometheus-cache-monitor1,6710312282ad4e0e,fe62a03f97135c59] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:23:06.355 INFO 26087 --- [prometheus-cache-monitor1,6710312282ad4e0e,fe62a03f97135c59] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:23:06.556 INFO 26087 --- [ocp-async-7,11de24b5205a1337,51ce4a9c07fb6ebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:23:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:23:06.623562+08:00, lastProcessTime=2025-04-08T21:22:56.624884+08:00 2025-04-08 21:23:06.632 WARN 26087 --- [ocp-async-4,58fd6bebc6863a19,f703dd416c923a8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:06.632 INFO 26087 --- [ocp-async-6,a9b6e981377e4da5,99afd569a4fa7e9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:06.696 INFO 26087 --- [ocp-schedules-26,d72247945b507247,aaedb25b6184b51c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:23:06.722 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:23:06.623562+08:00 2025-04-08 21:23:08.167 INFO 26087 --- [ocp-schedules-27,31f94a41fc17d99f,10865dea394d7ad6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:23:08.168 INFO 26087 --- [ocp-schedules-27,31f94a41fc17d99f,10865dea394d7ad6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:23:08.398 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:08.398 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:08.632 WARN 26087 --- [ocp-async-5,ec2b5d1b0a3d5b85,8ccde1ff662e657d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:08.632 INFO 26087 --- [ocp-async-10,d5ca7e968edc9802,fba34ffac34311c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:09.355 INFO 26087 --- [prometheus-minute-schedule-1,db968e498fcde9bd,fada5d51e41247fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:10.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:23:10.001 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:23:10.186 INFO 26087 --- [ocp-schedules-17,7a3fb4666513e6c4,595fb48ae30f91dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:23:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:23:10.632 WARN 26087 --- [ocp-async-8,3868ea385ea81afd,979e2bb290635593] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:10.632 INFO 26087 --- [ocp-async-2,f713ce697001ee59,8a26ea63a1ce8f47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:11.398 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:11.399 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:12.355 INFO 26087 --- [prometheus-minute-schedule-1,45549b110c878354,cfe0283d0385a419] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:12.435 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118520, nodeCount=17, affectRows=17 2025-04-08 21:23:12.436 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118520, nodeCount=17, validCount=17 2025-04-08 21:23:12.437 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118520, nodeCount=246, affectRows=246 2025-04-08 21:23:12.437 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118520, nodeCount=42, affectRows=42 2025-04-08 21:23:12.437 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118520, nodeCount=34, affectRows=34 2025-04-08 21:23:12.437 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118520, nodeCount=39, affectRows=39 2025-04-08 21:23:12.437 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118520, nodeCount=34, validCount=34 2025-04-08 21:23:12.437 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118520, nodeCount=42, validCount=42 2025-04-08 21:23:12.438 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118520, nodeCount=39, validCount=39 2025-04-08 21:23:12.438 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118520, nodeCount=245, validCount=245 2025-04-08 21:23:12.632 WARN 26087 --- [ocp-async-3,fd917334eb52b310,1210d1bf202bf399] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:12.632 INFO 26087 --- [ocp-async-4,a96d143d2bb7a075,fbe76eb63533e396] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:14.399 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:14.400 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:14.632 WARN 26087 --- [ocp-async-4,4b62e887b788e219,e3bf632c5303c5ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:14.632 INFO 26087 --- [ocp-async-5,0d4ffcd6ceb446e3,78feb1ce74bde598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:15.355 INFO 26087 --- [prometheus-minute-schedule-1,a0ac6c66fa856061,fd97808162198b2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:16.517 INFO 26087 --- [ocp-schedules-3,6f91a0dead1e69f7,c8f3ad9ccda87de1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:23:16.518 INFO 26087 --- [ocp-schedules-27,719abf858f2711bb,a0c4bc91ca3fc587] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:23:16.521 INFO 26087 --- [ocp-schedules-13,46e0034650cf0047,8a0abffe6b54d16f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:23:16.521 INFO 26087 --- [ocp-schedules-21,37d3412a1f917638,4b77005a886495b4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:23:16.521 INFO 26087 --- [ocp-schedules-21,37d3412a1f917638,4b77005a886495b4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:23:16.523 INFO 26087 --- [ocp-schedules-18,48f94c670111b95c,8c18c841d321317f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:23:16.523 INFO 26087 --- [ocp-schedules-18,48f94c670111b95c,8c18c841d321317f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:23:16.523 INFO 26087 --- [ocp-schedules-22,e2b90629e079b9c0,5013b327793be657] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:23:16.523 INFO 26087 --- [ocp-schedules-22,e2b90629e079b9c0,5013b327793be657] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:23:16.524 INFO 26087 --- [ocp-schedules-27,719abf858f2711bb,a0c4bc91ca3fc587] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:23:16.524 INFO 26087 --- [ocp-schedules-27,008cc35f7d2e0c35,9ef4fafb867fe68e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:23:16.524 INFO 26087 --- [ocp-schedules-8,594489d9a5aa7b97,5bccc0549b720ccf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:23:16.524 INFO 26087 --- [ocp-schedules-9,4aa89a19631ca255,dafd86865e07db7b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:23:16.534 INFO 26087 --- [ocp-schedules-3,6f91a0dead1e69f7,c8f3ad9ccda87de1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:23:16.536 INFO 26087 --- [ocp-schedules-29,c8ab4ab113fe8c39,c7a44589373d3eef] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:23:16.536 INFO 26087 --- [ocp-schedules-29,c8ab4ab113fe8c39,c7a44589373d3eef] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:23:16.537 INFO 26087 --- [ocp-schedules-15,3dd67309cfc52717,b36c9108e53b5a5b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:23:16.546 INFO 26087 --- [ocp-schedules-11,3bdfe52f515e724b,0638d6ac6b302968] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:23:16.554 INFO 26087 --- [pool-overview-container-2,046ad9334eca5dbe,8758f220772e65be] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:23:16.554 INFO 26087 --- [pool-overview-container-4,5d16c6097d169ed8,a590dac1c04ab70e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:23:16.556 INFO 26087 --- [ocp-async-8,feea4ca07c7b0b23,b39e40f06a1a7502] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:23:16.586 INFO 26087 --- [ocp-schedules-16,13c2161a725b1545,472c51c5ffb36326] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:23:16.586 INFO 26087 --- [ocp-schedules-16,13c2161a725b1545,472c51c5ffb36326] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:23:16.597 INFO 26087 --- [ocp-schedules-12,d94c6c7c840351af,1be6a8eb9fe3fba4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:23:16.601 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:23:16.601 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:23:16.605 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:23:16.605 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:23:16.605 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:23:16.606 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:23:16.606 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:23:16.606 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:23:16.608 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:23:16.608 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:23:16.608 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:23:16.609 INFO 26087 --- [ocp-schedules-4,e2644d0431b92314,ddcd13b2ce3b75b0] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:23:16.620 INFO 26087 --- [ocp-schedules-18,de04ecdad69f1d1a,5335485257aa762a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:23:16.621 INFO 26087 --- [ocp-schedules-19,8bfa732bb6a925ef,9cb847b845aa516b] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 21:23:16.621 INFO 26087 --- [ocp-schedules-19,8bfa732bb6a925ef,9cb847b845aa516b] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 21:23:16.621 INFO 26087 --- [ocp-schedules-19,8bfa732bb6a925ef,9cb847b845aa516b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=428 2025-04-08 21:23:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:23:16.623136+08:00, lastProcessTime=2025-04-08T21:23:06.623562+08:00 2025-04-08 21:23:16.631 INFO 26087 --- [ocp-schedules-22,e1da3f4c5fb1dbf4,17bdace7a29683e2] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:23:16.631 INFO 26087 --- [ocp-schedules-22,e1da3f4c5fb1dbf4,17bdace7a29683e2] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:23:16.632 WARN 26087 --- [ocp-async-1,6e80494014055aca,9f690873a86e7c05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:16.632 INFO 26087 --- [ocp-async-2,04a641970e1dca0f,893328f4b40bf151] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:16.637 INFO 26087 --- [ocp-schedules-19,8bfa732bb6a925ef,9cb847b845aa516b] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=428 2025-04-08 21:23:16.647 INFO 26087 --- [ocp-schedules-29,c536a84c510c6790,6f23ad56ef8843f5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:23:16.647+08:00, clusters=[] 2025-04-08 21:23:16.647 INFO 26087 --- [ocp-schedules-29,d1aefb01d70bb016,2718ebbb339edc69] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:23:16.647 INFO 26087 --- [ocp-schedules-29,d1aefb01d70bb016,2718ebbb339edc69] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:23:16.647 INFO 26087 --- [ocp-schedules-15,71c24be64499ca9f,53dc122ad391082b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:23:16.653 INFO 26087 --- [ocp-schedules-10,7dd986eae6825d7a,05f393c1367e6921] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:23:16.653+08:00, clusters=[] 2025-04-08 21:23:16.672 INFO 26087 --- [ocp-schedules-11,4cd602427c44a1c8,da838029b00b3f66] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:23:16.672 INFO 26087 --- [ocp-schedules-11,4cd602427c44a1c8,da838029b00b3f66] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:23:16.673 INFO 26087 --- [ocp-schedules-11,4cd602427c44a1c8,da838029b00b3f66] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:23:16.673 INFO 26087 --- [ocp-schedules-11,4cd602427c44a1c8,da838029b00b3f66] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:23:16.678 INFO 26087 --- [ocp-schedules-3,b59d7e0111e0225d,5579a6912de5db4b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:23:16.679 INFO 26087 --- [ocp-schedules-3,b59d7e0111e0225d,5579a6912de5db4b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:23:16.679 INFO 26087 --- [ocp-schedules-3,b59d7e0111e0225d,5579a6912de5db4b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:23:16.697 INFO 26087 --- [ocp-schedules-2,3da110408ef47754,f30cee37bdaa601e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:23:16.736 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:23:16.623136+08:00 2025-04-08 21:23:16.772 INFO 26087 --- [ocp-schedules-28,63e66ba6192c4a78,6121165a85b8dc3f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 21:23:16.796 INFO 26087 --- [ocp-schedules-23,cd72f39cf11abd50,35acf2fe98630372] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:23:16.824 INFO 26087 --- [ocp-schedules-5,a8f30ee6431693c6,6ccff22ffcfafb02] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:23:16.824 INFO 26087 --- [ocp-schedules-5,a8f30ee6431693c6,6ccff22ffcfafb02] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:23:16.825 INFO 26087 --- [ocp-schedules-5,a8f30ee6431693c6,6ccff22ffcfafb02] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:23:16.829 INFO 26087 --- [ocp-schedules-20,3e08d04a5d10d9d0,2afd4e2c3e9d4e97] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:23:16.832 INFO 26087 --- [ocp-schedules-1,28db4a3504463c53,eedb45309720bd08] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:23:16.843 INFO 26087 --- [ocp-schedules-25,d525b053f789454e,1367db9090670b00] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:23:16.856 INFO 26087 --- [ocp-schedules-30,3acb76c27cbc2c86,7ac1f0cb053b7ddb] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:23:16.856 INFO 26087 --- [ocp-schedules-30,3acb76c27cbc2c86,7ac1f0cb053b7ddb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:23:16.857 INFO 26087 --- [ocp-schedules-30,3acb76c27cbc2c86,7ac1f0cb053b7ddb] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:23:16.857 INFO 26087 --- [ocp-schedules-30,3acb76c27cbc2c86,7ac1f0cb053b7ddb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:23:16.857 INFO 26087 --- [ocp-schedules-30,3acb76c27cbc2c86,7ac1f0cb053b7ddb] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:23:16.857 INFO 26087 --- [ocp-schedules-30,3acb76c27cbc2c86,7ac1f0cb053b7ddb] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:23:17.157 INFO 26087 --- [ocp-schedules-4,2d12a7a7b5815c9e,9ea61e189755df2f] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:23:17.349 INFO 26087 --- [ocp-schedules-13,407d0fdf191453ca,ded5313227f241a2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:23:17.353 INFO 26087 --- [ocp-schedules-13,407d0fdf191453ca,ded5313227f241a2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:23:17.353 INFO 26087 --- [ocp-schedules-13,407d0fdf191453ca,ded5313227f241a2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:23:17.400 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:17.400 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:17.563 INFO 26087 --- [ocp-schedules-22,f83a89e957303e5f,8b213f46efdfb490] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:23:17.563 INFO 26087 --- [ocp-schedules-22,f83a89e957303e5f,8b213f46efdfb490] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:23:18.172 INFO 26087 --- [ocp-schedules-26,a1d3aa8dcb3af249,509dab5433eadc2c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:23:18.173 INFO 26087 --- [ocp-schedules-26,a1d3aa8dcb3af249,509dab5433eadc2c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:23:18.355 INFO 26087 --- [prometheus-minute-schedule-1,1799678378359085,487ab34b06183c75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:18.633 WARN 26087 --- [ocp-async-9,df49afc101b8b4eb,32f300d2192ddbf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:18.633 INFO 26087 --- [ocp-async-4,890c9ae4fdda51fe,c20cb52413fa719e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:20.203 INFO 26087 --- [ocp-schedules-28,c144e920b5003dab,778580cc4aae63f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:23:20.400 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:20.401 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:23:20.633 WARN 26087 --- [ocp-async-5,339037bdf7e8ed4f,a76a6ddb8e297caf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:20.633 INFO 26087 --- [ocp-async-8,85b5a50b8c668f48,2cdc634c76e0f162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:21.355 INFO 26087 --- [prometheus-minute-schedule-1,c5cd7e09407c68dc,fb4ede5e58616c67] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:22.633 INFO 26087 --- [ocp-async-1,4d976fe05d590c73,b3e0457f360edf33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:22.633 WARN 26087 --- [ocp-async-3,7f74483b773b9837,967704a797bc9463] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:23.401 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:23.401 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:24.355 INFO 26087 --- [prometheus-minute-schedule-1,658cb337e8766d8a,78010d897dd0f18d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:24.633 WARN 26087 --- [ocp-async-6,989f824e82d0a820,4bb6390a5867b9e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:24.633 INFO 26087 --- [ocp-async-9,fdd2abd51baa0428,73df1efdfba341d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:26.401 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:26.402 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:26.556 INFO 26087 --- [ocp-async-4,aa69a7bf4a6f7d38,eab99e3a9fcc868b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:23:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:23:26.622825+08:00, lastProcessTime=2025-04-08T21:23:16.623136+08:00 2025-04-08 21:23:26.633 WARN 26087 --- [ocp-async-8,7693b105d3f6778d,056726e7814df391] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:26.633 INFO 26087 --- [ocp-async-10,176c9d75283b8877,f576bd9163ee457b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:26.698 INFO 26087 --- [ocp-schedules-5,ddc36ddb54ae63b9,97e725eb05599e0a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:23:26.789 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:23:26.622825+08:00 2025-04-08 21:23:27.355 INFO 26087 --- [prometheus-minute-schedule-1,85b42c957e9ce34d,19509e892a502b50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:28.180 INFO 26087 --- [ocp-schedules-10,d9cb0014ba25d741,1482a07f39c1a2de] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:23:28.181 INFO 26087 --- [ocp-schedules-10,d9cb0014ba25d741,1482a07f39c1a2de] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:23:28.633 INFO 26087 --- [ocp-async-10,1a51fd4d4fd150fb,a55a4873a0f7ec64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:28.633 WARN 26087 --- [ocp-async-1,298ad6a78fa231b7,372a61ecf03c0909] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:29.402 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:29.403 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:30.223 INFO 26087 --- [ocp-schedules-12,cd0ad89ae946de5f,f177de1edf31351c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:23:30.355 INFO 26087 --- [prometheus-minute-schedule-1,887a215e961ba224,07cb020fa5244f6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:30.633 WARN 26087 --- [ocp-async-10,ef1eecd75af200a4,f33574cb8cb14964] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:30.634 INFO 26087 --- [ocp-async-9,2720517c6243caa0,44a110910b104fd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:32.403 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:32.403 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:32.634 WARN 26087 --- [ocp-async-3,7b47ce67b30e517f,187b55f888b54d6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:32.634 INFO 26087 --- [ocp-async-8,546e6f2d61fedc85,6449dbef6223860a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:33.355 INFO 26087 --- [prometheus-minute-schedule-1,a032da5353770b32,7c9de915ce830c54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:34.634 INFO 26087 --- [ocp-async-7,b119b0f0ff7c002a,6ccf01bedc5a0227] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:34.634 WARN 26087 --- [ocp-async-1,a7977be9f5ebc43b,e124522c7aa7bd32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:34.978 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:23:34.979 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:23:35.403 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:35.404 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:36.355 INFO 26087 --- [prometheus-minute-schedule-1,0bb4617db8e6c16e,35565bde1d0f70aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:36.556 INFO 26087 --- [ocp-async-2,b316c870d20302a1,fdd938d7aff8468f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:23:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:23:36.622953+08:00, lastProcessTime=2025-04-08T21:23:26.622825+08:00 2025-04-08 21:23:36.634 WARN 26087 --- [ocp-async-7,b3110cbe7556ec3c,fe4cd83a988ff902] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:36.634 INFO 26087 --- [ocp-async-10,83503b5b101ec6c8,7b1430abb6432c28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:36.699 INFO 26087 --- [ocp-schedules-30,c970f24b5a76536f,b741018cd73efe0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:23:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:23:36.622953+08:00 2025-04-08 21:23:37.354 INFO 26087 --- [ocp-schedules-19,3b69da3bcd1538e1,aba7286aeccdb0f1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:23:37.360 INFO 26087 --- [ocp-schedules-19,3b69da3bcd1538e1,aba7286aeccdb0f1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:23:37.360 INFO 26087 --- [ocp-schedules-19,3b69da3bcd1538e1,aba7286aeccdb0f1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:23:38.185 INFO 26087 --- [ocp-schedules-23,ed7b77ae0364e8e4,4d7b22675d65791b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:23:38.186 INFO 26087 --- [ocp-schedules-23,ed7b77ae0364e8e4,4d7b22675d65791b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:23:38.404 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:38.405 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:38.634 WARN 26087 --- [ocp-async-10,af7e920adb642942,bed34dc50678b259] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:38.634 INFO 26087 --- [ocp-async-4,d518a06beac18e5d,2536b5027668db75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:39.355 INFO 26087 --- [prometheus-minute-schedule-1,e6d875b999f0c1b3,915264db23d43fc2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:40.001 INFO 26087 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:23:40.001 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:23:40.239 INFO 26087 --- [ocp-schedules-26,9089e4f9739f14cf,99db6d1081810145] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:23:40.634 WARN 26087 --- [ocp-async-1,61e83d55754e900b,a8ed0dd1fe96eabc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:40.634 INFO 26087 --- [ocp-async-6,b53fd8ee9cf46e6b,12512b9c7a66c4a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:41.405 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:41.405 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:42.355 INFO 26087 --- [prometheus-minute-schedule-1,b73fda1d74a5d0dc,c5170d87984f34e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:42.634 WARN 26087 --- [ocp-async-9,cd715320062438a4,7b1ad02d6e300e2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:42.634 INFO 26087 --- [ocp-async-6,6e5321f8985ca0c9,3432655e03e3b249] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:44.405 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:44.406 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:44.634 WARN 26087 --- [ocp-async-3,9b4225b3e0cd4c85,81a9e4056c8315c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:44.634 INFO 26087 --- [ocp-async-10,dcbd145904f9610b,a06e2a1d07c00656] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:45.355 INFO 26087 --- [prometheus-minute-schedule-1,84a1e0cb707e3444,e4a86c093ea7318c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:46.521 INFO 26087 --- [ocp-schedules-18,6f559b2028b77832,d39fa186d172063e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:23:46.521 INFO 26087 --- [ocp-schedules-9,2b3bb7bdd6d22790,e355eada1f4fabe6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:23:46.521 INFO 26087 --- [ocp-schedules-9,2b3bb7bdd6d22790,e355eada1f4fabe6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:23:46.523 INFO 26087 --- [ocp-schedules-8,58fe75465358acad,3c4a56f1f9b1119a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:23:46.523 INFO 26087 --- [ocp-schedules-8,58fe75465358acad,3c4a56f1f9b1119a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:23:46.523 INFO 26087 --- [ocp-schedules-13,47ebeeec239b36d2,72c630749360f0fa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:23:46.523 INFO 26087 --- [ocp-schedules-13,47ebeeec239b36d2,72c630749360f0fa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:23:46.524 INFO 26087 --- [ocp-schedules-4,fed64fcc2df1ee42,f15ecaa9ccf2ef6b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:23:46.524 INFO 26087 --- [ocp-schedules-26,920a0db331046a14,ecc42a0e731b6ca6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:23:46.524 INFO 26087 --- [ocp-schedules-3,744b03d9010c37b8,b36e37fa1fa22059] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:23:46.554 INFO 26087 --- [ocp-schedules-17,9656d9f39ef1bfc9,448710b3f0648d08] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:23:46.557 INFO 26087 --- [ocp-async-1,b06c8752b2292303,8bc3b0ce44de2a2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:23:46.621 INFO 26087 --- [ocp-schedules-27,75d4830b96c0b75c,e2e905a554c3cdbc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:23:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:23:46.623028+08:00, lastProcessTime=2025-04-08T21:23:36.622953+08:00 2025-04-08 21:23:46.635 WARN 26087 --- [ocp-async-10,c44748153f5b1d17,03fe87e3017a9c18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:46.635 INFO 26087 --- [ocp-async-2,e81882639847ace7,c127c668c4a0ab82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:46.700 INFO 26087 --- [ocp-schedules-1,e6c29fd126b583a0,ea1f5cc9ffda95c4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:23:46.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:23:46.623028+08:00 2025-04-08 21:23:47.406 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:47.407 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:48.191 INFO 26087 --- [ocp-schedules-24,7610d0df5f12ea90,70f0a85ac143641a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:23:48.192 INFO 26087 --- [ocp-schedules-24,7610d0df5f12ea90,70f0a85ac143641a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:23:48.355 INFO 26087 --- [prometheus-minute-schedule-1,86d0cecd8d5d1c4f,1c4a425cd574ea56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:48.635 WARN 26087 --- [ocp-async-5,609f3068c87e1a25,e39dd0e6ea40c5d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:48.635 INFO 26087 --- [ocp-async-2,e6ac6de2ff6fef24,1786e1b87d319e41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:50.260 INFO 26087 --- [ocp-schedules-20,df6f7de7ee4a080e,b08b71de06f6adb9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:23:50.407 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:50.408 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:50.635 WARN 26087 --- [ocp-async-6,6cd46c41106662b1,d0f70f35a3648c65] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:50.635 INFO 26087 --- [ocp-async-2,6ce16041a9c629f3,1c4ae04750bdb4f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:51.355 INFO 26087 --- [prometheus-minute-schedule-1,1276cbf8e3a802cb,c927b9b90df0ff57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:52.635 WARN 26087 --- [ocp-async-10,9901a7e910dd0099,8656a124068aa397] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:52.635 INFO 26087 --- [ocp-async-2,c992d122182214b6,b8ae21ec76a416a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:53.408 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:53.408 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:54.355 INFO 26087 --- [prometheus-minute-schedule-1,be1971c0bc671f7b,dd0f96a03016e31b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:54.635 WARN 26087 --- [ocp-async-3,d5bd26025e0ec815,e9953e9d0b190566] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:54.635 INFO 26087 --- [ocp-async-2,2b6bcba80e640d3a,c743d7cddb14b28f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:56.408 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:56.409 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:23:56.557 INFO 26087 --- [ocp-async-7,9caf73d48fabaaf3,bce1aa2bbf80bf01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:23:56.572 INFO 26087 --- [ocp-schedules-26,90bd37fd31b9b248,70e2fe9d738b691b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:23:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:23:56.623735+08:00, lastProcessTime=2025-04-08T21:23:46.623028+08:00 2025-04-08 21:23:56.635 WARN 26087 --- [ocp-async-1,23f8a74ee8ec99ae,35aa719b80ddebdc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:56.635 INFO 26087 --- [ocp-async-8,16ee140c43652740,61ea1437a51897e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:56.701 INFO 26087 --- [ocp-schedules-19,6ff7ee5f63114c08,cd090620426b2714] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:23:56.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:23:56.623735+08:00 2025-04-08 21:23:57.355 INFO 26087 --- [prometheus-minute-schedule-1,ff14822923e7c8fa,cccac908c92b80f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118640, interval=60 2025-04-08 21:23:57.361 INFO 26087 --- [ocp-schedules-7,d3fd75aaed1d6402,4e082791faaec9f4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:23:57.368 INFO 26087 --- [ocp-schedules-7,d3fd75aaed1d6402,4e082791faaec9f4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:23:57.368 INFO 26087 --- [ocp-schedules-7,d3fd75aaed1d6402,4e082791faaec9f4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:23:58.196 INFO 26087 --- [ocp-schedules-15,b9174e364a4dc447,46118cc9907fd603] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:23:58.197 INFO 26087 --- [ocp-schedules-15,b9174e364a4dc447,46118cc9907fd603] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:23:58.635 WARN 26087 --- [ocp-async-4,d4acee75eeff9968,f260766cdb08cc59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:23:58.635 INFO 26087 --- [ocp-async-9,eb508749f6a58782,be6615a33cfa3f0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:23:59.409 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:23:59.409 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:00.296 INFO 26087 --- [ocp-schedules-25,300df8101ed29f5a,8e8e3be5368ef152] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:24:00.355 INFO 26087 --- [prometheus-minute-schedule-1,314bb796da6185ee,36282a0f88b7a223] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:00.636 WARN 26087 --- [ocp-async-3,362af4d7c137a61c,2e677e62a7f91d3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:00.636 INFO 26087 --- [ocp-async-5,2883ef93e092b604,6ff9128704e54182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:02.410 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:02.410 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:02.636 WARN 26087 --- [ocp-async-7,a0b4e33322e6ede5,a086505005d6199e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:02.636 INFO 26087 --- [ocp-async-1,be9075d321eaed9c,2fb553cd322c6b8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:03.355 INFO 26087 --- [prometheus-minute-schedule-1,97b27802b067cf6e,351ca096e97f3160] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:04.636 WARN 26087 --- [ocp-async-4,6d168e8467157304,8ca310827612e2d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:04.636 INFO 26087 --- [ocp-async-8,5631440b8f1ed316,3aa56d4d21869b8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:05.410 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:05.411 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:06.319 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:18:06.282, end=2025-04-08T21:23:06.314} 2025-04-08 21:24:06.321 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:24:06.321 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:24:06.324 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:18:06.292, end=2025-04-08T21:23:06.321} 2025-04-08 21:24:06.326 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:24:06.326 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:24:06.355 INFO 26087 --- [prometheus-minute-schedule-1,7861d5b67345bdf5,ff67c7143543ba0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:06.355 INFO 26087 --- [prometheus-cache-monitor1,d3d5649025d171fa,827aae41fa96d3a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:24:06.355 INFO 26087 --- [prometheus-cache-monitor1,d3d5649025d171fa,827aae41fa96d3a4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:24:06.502 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:20:06.456, end=2025-04-08T21:22:06.495} 2025-04-08 21:24:06.504 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:24:06.504 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:24:06.557 INFO 26087 --- [ocp-async-1,aa9836781544374f,2e2417bc42a0de7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:24:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:24:06.623669+08:00, lastProcessTime=2025-04-08T21:23:56.623735+08:00 2025-04-08 21:24:06.636 WARN 26087 --- [ocp-async-3,a66e815c1a0fd90e,0cab10097a002ed1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:06.636 INFO 26087 --- [ocp-async-5,4949920fbedeb5c1,5991b7b4c53024f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:06.703 INFO 26087 --- [ocp-schedules-20,58a2f218350fa3c5,b204cb5bd1a38f8c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:24:06.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:24:06.623669+08:00 2025-04-08 21:24:08.201 INFO 26087 --- [ocp-schedules-17,94638ee586b39520,c25d1c4417cd6848] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:24:08.203 INFO 26087 --- [ocp-schedules-17,94638ee586b39520,c25d1c4417cd6848] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:24:08.411 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:08.411 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:08.636 WARN 26087 --- [ocp-async-6,38e4686d25aca39f,f822bb8cfa07d97c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:08.636 INFO 26087 --- [ocp-async-10,99333133348903ef,c2c6dcc433eb81f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:09.355 INFO 26087 --- [prometheus-minute-schedule-1,79d3d856df9a2ff5,e19fea10915983d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:10.001 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:24:10.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:24:10.316 INFO 26087 --- [ocp-schedules-15,483889f03bb21693,52327e19eb1e70d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:24:10.601 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:24:10.636 WARN 26087 --- [ocp-async-10,fca599f22bc89655,d0b20dfc11548ab3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:10.636 INFO 26087 --- [ocp-async-1,58d93661075e9e8e,08d4c685b88f867f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:11.412 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:11.412 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:12.355 INFO 26087 --- [prometheus-minute-schedule-1,c8182a2deb9fde28,0a55c4f9c224c7d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:12.435 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118580, nodeCount=6, affectRows=6 2025-04-08 21:24:12.436 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118580, nodeCount=6, validCount=6 2025-04-08 21:24:12.440 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118580, nodeCount=96, affectRows=96 2025-04-08 21:24:12.440 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118580, nodeCount=127, affectRows=127 2025-04-08 21:24:12.440 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118580, nodeCount=149, affectRows=149 2025-04-08 21:24:12.441 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118580, nodeCount=96, validCount=96 2025-04-08 21:24:12.441 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118580, nodeCount=127, validCount=127 2025-04-08 21:24:12.442 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118580, nodeCount=148, validCount=148 2025-04-08 21:24:12.636 WARN 26087 --- [ocp-async-3,2c21de0aaa5da346,17e6d9e17784be8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:12.636 INFO 26087 --- [ocp-async-8,b9153be4b70b78ac,c32a94e4aaa7ffe0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:14.412 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:14.413 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:14.637 INFO 26087 --- [ocp-async-5,625e6fd3c6fcabdc,a18541d34e42a458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:14.637 WARN 26087 --- [ocp-async-7,ead4507a81437a3a,1be52d786061f812] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:15.355 INFO 26087 --- [prometheus-minute-schedule-1,0b16c12f83b05618,8aed9fb7878f7cb6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:16.517 INFO 26087 --- [ocp-schedules-12,11ef744a258c5971,2263e61691d40ced] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:24:16.517 INFO 26087 --- [ocp-schedules-4,11f99e4aacc9af71,f6ac7e7921d5189b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:24:16.521 INFO 26087 --- [ocp-schedules-30,2dda8887a347c37b,669aedab2289d614] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:24:16.521 INFO 26087 --- [ocp-schedules-30,2dda8887a347c37b,669aedab2289d614] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:24:16.521 INFO 26087 --- [ocp-schedules-30,bca85fd253be2b3e,a6010ec6f126ed42] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:24:16.523 INFO 26087 --- [ocp-schedules-23,a1ccce42df7378a2,d15bc4cb23d880d0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:24:16.523 INFO 26087 --- [ocp-schedules-23,a1ccce42df7378a2,d15bc4cb23d880d0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:24:16.523 INFO 26087 --- [ocp-schedules-21,2693471f7eac170a,75982b8eb401784f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:24:16.523 INFO 26087 --- [ocp-schedules-21,2693471f7eac170a,75982b8eb401784f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:24:16.524 INFO 26087 --- [ocp-schedules-20,7274e140859959a4,6d4b85a6fe21f7fe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:24:16.524 INFO 26087 --- [ocp-schedules-25,c94dd551a9d35e00,791324086253ee50] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:24:16.524 INFO 26087 --- [ocp-schedules-29,469264cfe10212da,b6a2f36f8db162d2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:24:16.526 INFO 26087 --- [ocp-schedules-4,11f99e4aacc9af71,f6ac7e7921d5189b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:24:16.538 INFO 26087 --- [ocp-schedules-12,11ef744a258c5971,2263e61691d40ced] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:24:16.548 INFO 26087 --- [ocp-schedules-11,c586ffe638211910,a37ba96bfe1a76d8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:24:16.557 INFO 26087 --- [ocp-async-10,ecbe2e37f55b7bb7,4dba62be3dfb7d39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:24:16.560 INFO 26087 --- [ocp-schedules-9,2a0d73655be30c01,ae7741c18acc66f6] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 21:24:16.560 INFO 26087 --- [ocp-schedules-9,2a0d73655be30c01,ae7741c18acc66f6] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 21:24:16.598 INFO 26087 --- [ocp-schedules-17,ee8a074743c75659,437d8d60ab721d31] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:24:16.622 INFO 26087 --- [ocp-schedules-24,2e6e331735e46612,853c95a6e283edb7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:24:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:24:16.623267+08:00, lastProcessTime=2025-04-08T21:24:06.623669+08:00 2025-04-08 21:24:16.633 INFO 26087 --- [ocp-schedules-3,cf27d463ef897bf9,3bc1a8d721f2adfc] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:24:16.633 INFO 26087 --- [ocp-schedules-3,cf27d463ef897bf9,3bc1a8d721f2adfc] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:24:16.637 WARN 26087 --- [ocp-async-3,7cd279755a5dd32d,c0446e1e6a3c252b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:16.637 INFO 26087 --- [ocp-async-2,4cb3f1b52082f19c,66177ba0d5a5e67d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:16.648 INFO 26087 --- [ocp-schedules-23,9289236097ff6c80,f8782b3c417c5c64] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:24:16.648+08:00, clusters=[] 2025-04-08 21:24:16.648 INFO 26087 --- [ocp-schedules-29,3e37fc5618e847f5,c406e9fd515e2bae] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:24:16.648 INFO 26087 --- [ocp-schedules-29,3e37fc5618e847f5,c406e9fd515e2bae] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:24:16.649 INFO 26087 --- [ocp-schedules-21,cf336dd80b8cdadf,d9947ee6c97a3c5e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:24:16.654 INFO 26087 --- [ocp-schedules-20,b585fdfd4005cb47,d955a3013bd2545e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:24:16.654+08:00, clusters=[] 2025-04-08 21:24:16.674 INFO 26087 --- [ocp-schedules-25,19c2d6cd7688d434,2f30f6c8027f99ad] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:24:16.674 INFO 26087 --- [ocp-schedules-25,19c2d6cd7688d434,2f30f6c8027f99ad] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:24:16.675 INFO 26087 --- [ocp-schedules-25,19c2d6cd7688d434,2f30f6c8027f99ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:24:16.675 INFO 26087 --- [ocp-schedules-25,19c2d6cd7688d434,2f30f6c8027f99ad] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:24:16.680 INFO 26087 --- [ocp-schedules-4,08118e8c28af67c3,05523d530f19493b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:24:16.681 INFO 26087 --- [ocp-schedules-4,08118e8c28af67c3,05523d530f19493b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:24:16.681 INFO 26087 --- [ocp-schedules-4,08118e8c28af67c3,05523d530f19493b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:24:16.704 INFO 26087 --- [ocp-schedules-12,395ea995e233dbd2,deac9df0529d3641] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:24:16.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:24:16.623267+08:00 2025-04-08 21:24:16.802 INFO 26087 --- [ocp-schedules-16,b41450c072e9924e,5049bb7f8d2ce98b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:24:16.830 INFO 26087 --- [ocp-schedules-14,07a32723debb9478,b07bf77bdd20780f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:24:16.844 INFO 26087 --- [ocp-schedules-9,ec7bc41c523db70f,abcf62115d6df3fc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:24:16.857 INFO 26087 --- [ocp-schedules-28,34856ffd71cf53ed,f8ddec7548f989ed] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:24:16.857 INFO 26087 --- [ocp-schedules-28,34856ffd71cf53ed,f8ddec7548f989ed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:24:16.858 INFO 26087 --- [ocp-schedules-28,34856ffd71cf53ed,f8ddec7548f989ed] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:24:16.858 INFO 26087 --- [ocp-schedules-28,34856ffd71cf53ed,f8ddec7548f989ed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:24:16.858 INFO 26087 --- [ocp-schedules-28,34856ffd71cf53ed,f8ddec7548f989ed] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:24:16.858 INFO 26087 --- [ocp-schedules-28,34856ffd71cf53ed,f8ddec7548f989ed] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:24:17.369 INFO 26087 --- [ocp-schedules-7,b25172dc2abc553c,9e907fa1405b890a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:24:17.373 INFO 26087 --- [ocp-schedules-7,b25172dc2abc553c,9e907fa1405b890a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:24:17.373 INFO 26087 --- [ocp-schedules-7,b25172dc2abc553c,9e907fa1405b890a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:24:17.413 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:17.414 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:17.565 INFO 26087 --- [ocp-schedules-3,7b71e14e2a7b04be,b3369eebf726b6f3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:24:17.566 INFO 26087 --- [ocp-schedules-3,7b71e14e2a7b04be,b3369eebf726b6f3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:24:18.213 INFO 26087 --- [ocp-schedules-25,c5503938bf84847a,f9dc7bd1aa2142f9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:24:18.214 INFO 26087 --- [ocp-schedules-25,c5503938bf84847a,f9dc7bd1aa2142f9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:24:18.355 INFO 26087 --- [prometheus-minute-schedule-1,1ee7933272a7c650,1d0f2b75e9139671] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:18.637 WARN 26087 --- [ocp-async-7,fa848239cdb9ea07,0b91851cc0127a8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:18.637 INFO 26087 --- [ocp-async-6,a20f94d6c17b74c6,070a371ba41ebb89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:20.332 INFO 26087 --- [ocp-schedules-18,35b380fe1d622148,444d0d2476c08d86] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:24:20.414 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:20.414 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:24:20.637 WARN 26087 --- [ocp-async-5,bfdce93db29cfe61,cc87f3647d6824a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:20.637 INFO 26087 --- [ocp-async-6,7efc587309d1386d,63fe2d1f90c2268b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:21.355 INFO 26087 --- [prometheus-minute-schedule-1,6c16b60bac95451d,839fa2630151a12e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:22.637 WARN 26087 --- [ocp-async-3,7d0a6ef31d2efe63,03a75e3b11478df0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:22.637 INFO 26087 --- [ocp-async-9,89186109431a51d8,b4b1431a6498dd66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:23.414 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:23.415 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:24.355 INFO 26087 --- [prometheus-minute-schedule-1,08fc3b338591ffe4,73b7bb21695d4a8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:24.637 WARN 26087 --- [ocp-async-9,2eed4d2a20c2ace3,29ed9d816db5a85d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:24.637 INFO 26087 --- [ocp-async-8,89b99dfc5b751466,8657f54b39611cde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:25.260 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:24:25.260 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:24:25.263 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:24:25.263 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:24:25.263 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:24:25.263 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:24:25.267 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:24:25.267 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:24:25.267 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:24:25.267 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:24:25.289 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:24:25.294 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:24:25.294 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:24:25.294 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:24:25.295 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:24:25.295 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:24:25.300 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:24:25.594 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:24:25.914 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.647705615S 2025-04-08 21:24:25.915 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:24:25.969 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:24:25.969 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:24:25.991 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:24:26.029 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:24:26.029 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:24:26.029 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:24:26.029 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:24:26.029 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:24:26.029 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:24:26.036 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:24:26.135 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] o.s.boot.SpringApplication : Started application in 0.871 seconds (JVM running for 1371.508) 2025-04-08 21:24:26.135 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.872093279S 2025-04-08 21:24:26.386 INFO 26087 --- [ocp-schedules-1,c6009e033200533b,d95a855a753f3246] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:24:26.415 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:26.416 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:26.557 INFO 26087 --- [ocp-async-4,58042a996df5cd3b,167e42de7d529587] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:24:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:24:26.623146+08:00, lastProcessTime=2025-04-08T21:24:16.623267+08:00 2025-04-08 21:24:26.637 WARN 26087 --- [ocp-async-10,e4b839782c33c9a8,b5b33af425dbc3de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:26.637 INFO 26087 --- [ocp-async-3,2afb8d79b91a5b75,003477baf614fc2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:26.705 INFO 26087 --- [ocp-schedules-27,acf4c4aa152e5294,21b1efc7372c0992] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:24:26.742 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:24:26.623146+08:00 2025-04-08 21:24:27.355 INFO 26087 --- [prometheus-minute-schedule-1,582291ee9da9f8e1,96ecdfc869570e32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:28.224 INFO 26087 --- [ocp-schedules-28,f9855443cfaab74f,df562d604b4da52b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:24:28.226 INFO 26087 --- [ocp-schedules-28,f9855443cfaab74f,df562d604b4da52b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:24:28.638 WARN 26087 --- [ocp-async-9,120e2057ad32e276,97f66419f02da451] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:28.638 INFO 26087 --- [ocp-async-2,0b1137876c13b36f,6a5efe3f0b39d3b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:29.416 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:29.417 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:30.355 INFO 26087 --- [prometheus-minute-schedule-1,30615ab41b9cdf66,49a1ab883af6b93f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:30.362 INFO 26087 --- [ocp-schedules-13,c534bf8c48b07758,ae9802197647cd1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:24:30.638 WARN 26087 --- [ocp-async-8,c2ac4fcafa5e47f0,21e3c09e83365157] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:30.638 INFO 26087 --- [ocp-async-7,55d118cab6285716,8c8e0359d49c8a07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:32.417 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:32.417 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:32.638 WARN 26087 --- [ocp-async-10,e8665cfb5dee8a3b,5b1a5a31c7a1eb98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:32.638 INFO 26087 --- [ocp-async-4,33f5177587a183a2,3b9d5b9217fd8b84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:33.355 INFO 26087 --- [prometheus-minute-schedule-1,26eb0f680730c81e,5e4b2b55f7454869] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:34.638 WARN 26087 --- [ocp-async-3,d9ee918ba3e1c2c7,edd82447faa1de17] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:34.638 INFO 26087 --- [ocp-async-9,76648af7c47da081,f5992aa7fa60af50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:35.417 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:35.418 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:36.355 INFO 26087 --- [prometheus-minute-schedule-1,75790369dcc7f572,8b2a6bd9f040a58f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:36.557 INFO 26087 --- [ocp-async-9,38294a8d2d69de7c,2465adc4b4aa3b53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:24:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:24:36.623904+08:00, lastProcessTime=2025-04-08T21:24:26.623146+08:00 2025-04-08 21:24:36.638 WARN 26087 --- [ocp-async-5,67a121a45be89654,8e049f0d5109fd15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:36.638 INFO 26087 --- [ocp-async-7,4be2816420c1a151,1d1c0535be7a9f61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:36.703 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:24:36.623904+08:00 2025-04-08 21:24:36.706 INFO 26087 --- [ocp-schedules-16,980507f661f3d1a2,1cee6b250cf86c7d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:24:37.375 INFO 26087 --- [ocp-schedules-6,e8cf9df07a89e8ad,ad73737bb7f37bbb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:24:37.379 INFO 26087 --- [ocp-schedules-6,e8cf9df07a89e8ad,ad73737bb7f37bbb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:24:37.379 INFO 26087 --- [ocp-schedules-6,e8cf9df07a89e8ad,ad73737bb7f37bbb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:24:38.229 INFO 26087 --- [ocp-schedules-15,ada286f798c364a2,c9f3a5b8425a6e45] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:24:38.230 INFO 26087 --- [ocp-schedules-15,ada286f798c364a2,c9f3a5b8425a6e45] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:24:38.418 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:38.418 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:38.638 WARN 26087 --- [ocp-async-7,3ab9e1cbae9b8edd,d08c499f7393a041] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:38.638 INFO 26087 --- [ocp-async-6,75a0416eb7710b8e,8c3f6aeaa24e3569] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:39.355 INFO 26087 --- [prometheus-minute-schedule-1,5f9808f2684b9703,720bb150ef4e8941] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:40.002 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:24:40.002 INFO 26087 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:24:40.380 INFO 26087 --- [ocp-schedules-18,bf21ca0690fe2638,614d1f47e43a1d48] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:24:40.638 WARN 26087 --- [ocp-async-3,20e1f83503e9e5f3,859f42973949f024] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:40.638 INFO 26087 --- [ocp-async-1,f201c86205b4c3aa,d09fcc86231746a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:41.419 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:41.419 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:42.355 INFO 26087 --- [prometheus-minute-schedule-1,e043fb2fb07497ed,116a8a7f5f37ab4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:42.639 WARN 26087 --- [ocp-async-1,90f2f6df5f87d3c0,44df7ddd8c4f017b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:42.639 INFO 26087 --- [ocp-async-8,9f5b397e1383502a,e7ede235ee3147f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:44.419 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:44.420 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:44.639 WARN 26087 --- [ocp-async-7,2946bbe791707954,ba2d1a5b1cbde4fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:44.639 INFO 26087 --- [ocp-async-4,2c9f9132f35fce69,2e637a1ff3fd3708] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:45.355 INFO 26087 --- [prometheus-minute-schedule-1,d675a437f6369c3c,73e1f99e3c68a6ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:46.521 INFO 26087 --- [ocp-schedules-5,fb68f0a3a797fec3,31631e077cd35cff] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:24:46.521 INFO 26087 --- [ocp-schedules-30,ace9a01b5c76cc65,ff6ea93b76e0ecd6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:24:46.521 INFO 26087 --- [ocp-schedules-30,ace9a01b5c76cc65,ff6ea93b76e0ecd6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:24:46.523 INFO 26087 --- [ocp-schedules-3,77613eaa892b61a0,fad668159555d447] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:24:46.523 INFO 26087 --- [ocp-schedules-3,77613eaa892b61a0,fad668159555d447] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:24:46.523 INFO 26087 --- [ocp-schedules-16,59a571117caa4b80,7692c88806016ded] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:24:46.523 INFO 26087 --- [ocp-schedules-16,59a571117caa4b80,7692c88806016ded] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:24:46.524 INFO 26087 --- [ocp-schedules-23,eb9cec6a19aa318e,04237c2557cc39cf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:24:46.524 INFO 26087 --- [ocp-schedules-15,ec2318bf92f6e4e2,fef87e52aa19420c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:24:46.524 INFO 26087 --- [ocp-schedules-24,873c2118e175f264,2043be2d473d058b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:24:46.524 INFO 26087 --- [ocp-schedules-26,0e14b235b8e6221e,729817ee9e1dfd74] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:24:46.557 INFO 26087 --- [ocp-async-4,8d9d2ccf0e07bed0,50078e4756f4de4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:24:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:24:46.623217+08:00, lastProcessTime=2025-04-08T21:24:36.623904+08:00 2025-04-08 21:24:46.623 INFO 26087 --- [ocp-schedules-17,bb5b377b178f909a,8f61fef99977dd97] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:24:46.639 WARN 26087 --- [ocp-async-3,57b9a685fc5713e6,e784e20470e833b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:46.639 INFO 26087 --- [ocp-async-6,303474e321c8cfcd,8e5dcfdf9d8aef38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:46.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:24:46.623217+08:00 2025-04-08 21:24:46.707 INFO 26087 --- [ocp-schedules-2,86dec40b97875283,b86954253ebab768] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:24:47.420 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:47.420 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:48.234 INFO 26087 --- [ocp-schedules-7,0ebeb93f5e29344e,90a9af724b22a0be] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:24:48.235 INFO 26087 --- [ocp-schedules-7,0ebeb93f5e29344e,90a9af724b22a0be] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:24:48.355 INFO 26087 --- [prometheus-minute-schedule-1,cb1496778e269429,119079052ddf8cbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:48.639 WARN 26087 --- [ocp-async-6,15c0334ecc8c7fb1,2d38a4e43bd4bec6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:48.639 INFO 26087 --- [ocp-async-2,1226cb47200539de,47512bd1eb26f93c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:50.399 INFO 26087 --- [ocp-schedules-7,2d602bc46f426049,9107da35992822f2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:24:50.420 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:50.421 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:50.639 WARN 26087 --- [ocp-async-7,c928ca0c915ede1b,89cdcc16e30c8444] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:50.639 INFO 26087 --- [ocp-async-8,deafc737a882de79,4bd09d0bd5daf837] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:51.355 INFO 26087 --- [prometheus-minute-schedule-1,70ad4bf01ce53b57,fe46dc93220ff04c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:52.639 WARN 26087 --- [ocp-async-3,582d255a68dd273a,cf3703a7ac01ecfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:52.639 INFO 26087 --- [ocp-async-9,275dbc02c902827c,5070346c29890229] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:53.421 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:53.422 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:54.355 INFO 26087 --- [prometheus-minute-schedule-1,b58041066d4bd7c3,9b1a79a77dec624d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:54.639 WARN 26087 --- [ocp-async-9,8b1d688c635ec903,c80e620ec83900d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:54.639 INFO 26087 --- [ocp-async-6,34b6a4d1781af0b4,f3fde6a5824b7375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:56.422 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:56.422 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:24:56.558 INFO 26087 --- [ocp-async-6,443c60324c33512a,0c054fd666d1a265] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:24:56.573 INFO 26087 --- [ocp-schedules-18,f259ff0590916510,53c488af613941e7] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:24:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:24:56.622641+08:00, lastProcessTime=2025-04-08T21:24:46.623217+08:00 2025-04-08 21:24:56.640 WARN 26087 --- [ocp-async-10,3383671956527dad,7422e68c3980cf46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:56.640 INFO 26087 --- [ocp-async-7,9b66b88faee6c019,05dfe47371f791f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:56.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:24:56.622641+08:00 2025-04-08 21:24:56.708 INFO 26087 --- [ocp-schedules-8,ed23bc0dd9c3dcbf,31fbad58fb367364] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:24:57.355 INFO 26087 --- [prometheus-minute-schedule-1,6247d0c60e08991a,bd510c3a33488665] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118700, interval=60 2025-04-08 21:24:57.380 INFO 26087 --- [ocp-schedules-28,102ba0127fdfcf83,3e1d7eaa04ba4bcf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:24:57.386 INFO 26087 --- [ocp-schedules-28,102ba0127fdfcf83,3e1d7eaa04ba4bcf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:24:57.386 INFO 26087 --- [ocp-schedules-28,102ba0127fdfcf83,3e1d7eaa04ba4bcf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:24:58.249 INFO 26087 --- [ocp-schedules-24,42613d88c921958f,ba5199215f660ae0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:24:58.250 INFO 26087 --- [ocp-schedules-24,42613d88c921958f,ba5199215f660ae0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:24:58.640 WARN 26087 --- [ocp-async-4,15cb18fcb070a5b9,90fb2206ba2e6c52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:24:58.640 INFO 26087 --- [ocp-async-3,438f6729e14f9072,ad13c0dd6d7ff642] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:24:59.422 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:24:59.423 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:00.355 INFO 26087 --- [prometheus-minute-schedule-1,8a068adbdad662ce,280f780f902d1484] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:00.416 INFO 26087 --- [ocp-schedules-16,efbc4fa3d981e544,7458b520795df5c4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:25:00.640 WARN 26087 --- [ocp-async-5,d9e72be3b1e3f1b9,aac9c18d46b7a454] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:00.640 INFO 26087 --- [ocp-async-2,4fb1d4a603ee7405,d6c0715ea2a1db4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:00.929 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:20:00.0, nextRun=2025-04-08 21:25:00.0, currentDate=Tue Apr 08 21:25:00 CST 2025 2025-04-08 21:25:00.929 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:20:00.0, nextRunTime=2025-04-08 21:25:00.0} 2025-04-08 21:25:00.932 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:25:00.932 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:25:00.932+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:25:00.942 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:20:00.0, nextRun=2025-04-08 21:25:00.0, currentDate=Tue Apr 08 21:25:00 CST 2025 2025-04-08 21:25:00.942 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:20:00.0, nextRunTime=2025-04-08 21:25:00.0} 2025-04-08 21:25:00.943 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:25:00.961 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:25:00.961 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,ee7ec82567fbc2e5] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:25:01.850 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,55b1ac8fbe7ffd99] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000122, operation:EXECUTE, state: RUNNING 2025-04-08 21:25:01.866 INFO 26087 --- [subtask-executor10,8ec1a44aeefd3a62,ed5e634948d136aa] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:25:01.867 INFO 26087 --- [subtask-executor10,8ec1a44aeefd3a62,ed5e634948d136aa] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000122, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000013, task_operation=execute, latest_execution_start_time=2025-04-08T21:25:01.846+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000122}, listMap={}}, executor=192.168.1.61 2025-04-08 21:25:01.875 INFO 26087 --- [subtask-executor10,8ec1a44aeefd3a62,ed5e634948d136aa] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000122, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:25:02.361 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000013, status:SUCCESS 2025-04-08 21:25:02.362 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000013, status:SUCCESS 2025-04-08 21:25:02.423 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:02.424 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:02.640 WARN 26087 --- [ocp-async-6,97dec2b328245926,d603d9d0ccc31b0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:02.640 INFO 26087 --- [ocp-async-10,77992b4893503fb8,78220ff3e867f93b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:03.355 INFO 26087 --- [prometheus-minute-schedule-1,88b7891dc1c9484a,39777707fb3e0bbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:04.640 WARN 26087 --- [ocp-async-4,e604154383051417,6273ecdb94af37fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:04.640 INFO 26087 --- [ocp-async-7,0cfb97258d47e03d,eb0ee1df5b5e4fb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:05.424 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:05.424 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:06.355 INFO 26087 --- [prometheus-minute-schedule-1,6c523802b28ee1f5,a3e7692b4dd5f231] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:06.355 INFO 26087 --- [prometheus-cache-monitor1,9bfd90b63bbdb46f,d5c2ff52a5a8e96f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:25:06.355 INFO 26087 --- [prometheus-cache-monitor1,9bfd90b63bbdb46f,d5c2ff52a5a8e96f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:25:06.558 INFO 26087 --- [ocp-async-7,47b4f678832fe2ee,75987c55f141a60a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:25:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:25:06.623872+08:00, lastProcessTime=2025-04-08T21:24:56.622641+08:00 2025-04-08 21:25:06.640 WARN 26087 --- [ocp-async-5,3e41c03424c625d7,51d815d14406e712] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:06.640 INFO 26087 --- [ocp-async-9,a34f37d31debc0c1,0f43e8111e900b35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:06.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:25:06.623872+08:00 2025-04-08 21:25:06.709 INFO 26087 --- [ocp-schedules-26,27c7e804ccf81a52,b2757894632a1fb8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:25:08.256 INFO 26087 --- [ocp-schedules-7,8df1a0b4487b7af7,78cee19a238318fb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:25:08.257 INFO 26087 --- [ocp-schedules-7,8df1a0b4487b7af7,78cee19a238318fb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:25:08.425 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:08.425 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:08.640 WARN 26087 --- [ocp-async-8,8adc7615efd0383e,dc71c548fa85f524] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:08.640 INFO 26087 --- [ocp-async-6,3a9a85eeff942959,5469b482cdbcc342] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:09.355 INFO 26087 --- [prometheus-minute-schedule-1,3e7a26d7bc839ddf,434a5f203ad1e521] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:10.001 INFO 26087 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:25:10.001 INFO 26087 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:25:10.452 INFO 26087 --- [ocp-schedules-21,291ef01302bd09b2,99732d735fac9b3f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:25:10.604 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:25:10.641 WARN 26087 --- [ocp-async-1,ac8b06c137c287d2,e14b2e5b94146b7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:10.641 INFO 26087 --- [ocp-async-4,a7215014a6d324ad,625ff4ba91ba29fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:11.425 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:11.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:12.355 INFO 26087 --- [prometheus-minute-schedule-1,74178ceb7941bf4b,ab04bd809fd2f4fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:12.453 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118640, nodeCount=42, affectRows=42 2025-04-08 21:25:12.453 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118640, nodeCount=42, validCount=42 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118640, nodeCount=53, affectRows=53 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118640, nodeCount=65, affectRows=65 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118640, nodeCount=45, affectRows=45 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118640, nodeCount=88, affectRows=88 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118640, nodeCount=85, affectRows=85 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118640, nodeCount=53, validCount=53 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118640, nodeCount=45, validCount=45 2025-04-08 21:25:12.468 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118640, nodeCount=65, validCount=65 2025-04-08 21:25:12.469 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118640, nodeCount=87, validCount=87 2025-04-08 21:25:12.469 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118640, nodeCount=85, validCount=85 2025-04-08 21:25:12.641 WARN 26087 --- [ocp-async-7,0b7f4dcc1169a167,1c4323dad856a307] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:12.641 INFO 26087 --- [ocp-async-2,d0bcde58e79dda27,1a83a9df479deddf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:14.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:14.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:14.641 WARN 26087 --- [ocp-async-9,239a1e834004e2a9,03d77d6929db8c83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:14.641 INFO 26087 --- [ocp-async-8,ba0ee0b28ad6c2cb,c29695cea63dc32b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:15.355 INFO 26087 --- [prometheus-minute-schedule-1,6962e0c038e5f277,88e5a8d9ed4826f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:16.517 INFO 26087 --- [ocp-schedules-18,fa3bd6d25518aca2,ed00dbe51777b738] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:25:16.517+08:00, tenantSize=0 2025-04-08 21:25:16.517 INFO 26087 --- [ocp-schedules-18,51ab5e85fb1263a2,e4114853fdba4391] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:25:16.517 INFO 26087 --- [ocp-schedules-9,1594ebfb31883a02,be964b1bc27c09e3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:25:16.521 INFO 26087 --- [ocp-schedules-29,63f562fa7fefadcd,f4503388d82c3742] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:25:16.521 INFO 26087 --- [ocp-schedules-28,8749d5a02829595c,6d457166568375a8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:25:16.521 INFO 26087 --- [ocp-schedules-28,8749d5a02829595c,6d457166568375a8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:25:16.523 INFO 26087 --- [ocp-schedules-10,f82993139d0a3a31,61e8c4af2103e85d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:25:16.523 INFO 26087 --- [ocp-schedules-10,f82993139d0a3a31,61e8c4af2103e85d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:25:16.523 INFO 26087 --- [ocp-schedules-23,5ff3c4faaa7f5a52,fc545a34cb1b4df5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:25:16.523 INFO 26087 --- [ocp-schedules-23,5ff3c4faaa7f5a52,fc545a34cb1b4df5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:25:16.523 INFO 26087 --- [ocp-schedules-9,1594ebfb31883a02,be964b1bc27c09e3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:25:16.524 INFO 26087 --- [ocp-schedules-16,d259b853f0ca251a,a6f60801a15db8a6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:25:16.524 INFO 26087 --- [ocp-schedules-5,3e658da00f773ebc,c505f79cd4cbed52] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:25:16.524 INFO 26087 --- [ocp-schedules-11,9bc758e41c36236d,5f7bf9aab1fb8f65] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:25:16.526 INFO 26087 --- [ocp-schedules-18,51ab5e85fb1263a2,e4114853fdba4391] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:25:16.529 INFO 26087 --- [ocp-schedules-12,f575807dfa2dc877,d4562b908e981767] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:25:16.558 INFO 26087 --- [ocp-async-1,13858e770b2cbfc4,72c6af2263d9fd28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:25:16.561 INFO 26087 --- [ocp-schedules-7,664fa5030b1867dd,5a88c8aeb494ff7d] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 21:25:16.561 INFO 26087 --- [ocp-schedules-7,664fa5030b1867dd,5a88c8aeb494ff7d] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 21:25:16.561 INFO 26087 --- [ocp-schedules-26,bbcb24cd8c4128df,18a3bb5a438d90e2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:25:16.562 INFO 26087 --- [ocp-schedules-26,bbcb24cd8c4128df,18a3bb5a438d90e2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:25:16.586 INFO 26087 --- [ocp-schedules-24,e1e7a7119327d0e4,e9aaecbda25a1e54] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:25:16.586 INFO 26087 --- [ocp-schedules-24,e1e7a7119327d0e4,e9aaecbda25a1e54] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:25:16.600 INFO 26087 --- [ocp-schedules-25,f724a47ccd204d29,af0a398a54939fca] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:25:16.624 INFO 26087 --- [ocp-schedules-17,dac2ea2f48ee6a30,4cac01fd97b25b96] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:25:16.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:25:16.624917+08:00, lastProcessTime=2025-04-08T21:25:06.623872+08:00 2025-04-08 21:25:16.635 INFO 26087 --- [ocp-schedules-28,721f9ea565bbce4e,b1cecd3df1ac6914] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:25:16.635 INFO 26087 --- [ocp-schedules-28,721f9ea565bbce4e,b1cecd3df1ac6914] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:25:16.641 WARN 26087 --- [ocp-async-7,f7e5e2d77884efc8,f646e040c0942d7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:16.641 INFO 26087 --- [ocp-async-4,3703863ef03b97e4,e7bceae05cbe2448] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:16.650 INFO 26087 --- [ocp-schedules-8,670881b9d6351e8d,e9c04e12c7b1ea36] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:25:16.649+08:00, clusters=[] 2025-04-08 21:25:16.650 INFO 26087 --- [ocp-schedules-5,c6b896b06e50ebac,7381e07444ba0b50] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:25:16.650 INFO 26087 --- [ocp-schedules-5,c6b896b06e50ebac,7381e07444ba0b50] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:25:16.650 INFO 26087 --- [ocp-schedules-9,776ee26eb48bd4c5,5e688f018a14e54d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:25:16.655 INFO 26087 --- [ocp-schedules-16,82606a36f3763e41,2cb80310462cccf0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:25:16.655+08:00, clusters=[] 2025-04-08 21:25:16.675 INFO 26087 --- [ocp-schedules-11,d44dbc31b8157af5,91d6991b22f282c9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:25:16.675 INFO 26087 --- [ocp-schedules-11,d44dbc31b8157af5,91d6991b22f282c9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:25:16.676 INFO 26087 --- [ocp-schedules-11,d44dbc31b8157af5,91d6991b22f282c9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:25:16.676 INFO 26087 --- [ocp-schedules-11,d44dbc31b8157af5,91d6991b22f282c9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:25:16.681 INFO 26087 --- [ocp-schedules-12,5d4f646f44757c1c,08f9b25810f489fd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:25:16.682 INFO 26087 --- [ocp-schedules-12,5d4f646f44757c1c,08f9b25810f489fd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:25:16.682 INFO 26087 --- [ocp-schedules-12,5d4f646f44757c1c,08f9b25810f489fd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:25:16.710 INFO 26087 --- [ocp-schedules-22,7942b098c505818a,2d3df212ca9ff4d5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:25:16.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:25:16.624917+08:00 2025-04-08 21:25:16.815 INFO 26087 --- [ocp-schedules-21,52e36c26720bf164,7e33709c5d697e91] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:25:16.831 INFO 26087 --- [ocp-schedules-7,cd9c2ba4101081df,2512f6d4782a560f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:25:16.832 INFO 26087 --- [ocp-schedules-26,1997fe76fb614658,9eb6ff1fb246486a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:25:16.845 INFO 26087 --- [ocp-schedules-6,5c75f38fc51c23b9,627ee586279695eb] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:25:16.859 INFO 26087 --- [ocp-schedules-14,abe33cf89d8cb5d8,5697f191eacc11c4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:25:16.859 INFO 26087 --- [ocp-schedules-14,abe33cf89d8cb5d8,5697f191eacc11c4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:25:16.862 INFO 26087 --- [ocp-schedules-14,abe33cf89d8cb5d8,5697f191eacc11c4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:25:16.862 INFO 26087 --- [ocp-schedules-14,abe33cf89d8cb5d8,5697f191eacc11c4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:25:16.864 INFO 26087 --- [ocp-schedules-14,abe33cf89d8cb5d8,5697f191eacc11c4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:25:16.864 INFO 26087 --- [ocp-schedules-14,abe33cf89d8cb5d8,5697f191eacc11c4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:25:17.164 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:25:17.164 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:25:17.165 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.165 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:25:17.166 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.166 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.166 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.166 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:25:17.179 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.179 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.180 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.180 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:25:17.190 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.190 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.190 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.191 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:25:17.201 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.201 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.202 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.202 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:25:17.212 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.213 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.213 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.213 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:25:17.218 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.218 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.218 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.219 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:25:17.222 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.222 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.223 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.223 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:25:17.228 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.228 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.228 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.228 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:25:17.233 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.233 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.233 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.233 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:25:17.237 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.237 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.238 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.238 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:25:17.242 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.242 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.242 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.242 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:25:17.249 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.249 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.249 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.250 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:25:17.261 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.261 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.261 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.261 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:25:17.265 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.266 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.266 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.266 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:25:17.271 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.271 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.272 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.272 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:25:17.276 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.276 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.276 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.276 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:25:17.280 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.280 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.280 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.280 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:25:17.283 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.284 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.284 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.284 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:25:17.287 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.287 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.288 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.288 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:25:17.291 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.291 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.291 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.292 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:25:17.295 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.295 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.295 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.295 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:25:17.298 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.299 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.299 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.299 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:25:17.302 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.302 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.302 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.303 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:25:17.306 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.306 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.306 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.306 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:25:17.309 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.310 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.310 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.310 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:25:17.313 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.313 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.313 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.313 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:25:17.316 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.316 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.317 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.317 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:25:17.320 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.320 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.320 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.321 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:25:17.324 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.324 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.325 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.325 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:25:17.328 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.328 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.329 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.329 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:25:17.332 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.332 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.332 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.332 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:25:17.335 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.336 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.336 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.336 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:25:17.339 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.340 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.340 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.340 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:25:17.343 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.343 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.344 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.344 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:25:17.351 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.351 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.352 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.352 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:25:17.357 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.357 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.357 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.357 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:25:17.362 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.362 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.363 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.363 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:25:17.368 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.368 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.368 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.369 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:25:17.372 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.372 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.372 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.372 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:25:17.375 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.375 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.376 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.376 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:25:17.379 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.379 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.379 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.379 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:25:17.382 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.382 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.382 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.382 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:25:17.387 INFO 26087 --- [ocp-schedules-25,d17822264b68cfa0,7717345eb45997cf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:25:17.390 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.390 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.390 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.391 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:25:17.391 INFO 26087 --- [ocp-schedules-25,d17822264b68cfa0,7717345eb45997cf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:25:17.391 INFO 26087 --- [ocp-schedules-25,d17822264b68cfa0,7717345eb45997cf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:25:17.408 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.408 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.408 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.408 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:25:17.413 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.413 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.413 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.414 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:25:17.418 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.418 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.418 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.419 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:25:17.423 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.423 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.424 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.424 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:25:17.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:17.427 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:17.427 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.427 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.428 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.428 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:25:17.433 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.433 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.433 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.433 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:25:17.437 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:25:17.437 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:25:17.437 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.438 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:25:17.438 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.438 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.439 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.439 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:25:17.443 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.443 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:25:17.443 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:17.443 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:25:17.446 INFO 26087 --- [ocp-schedules-20,05717fad68ae4dd0,c09cd163c72600a9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:25:17.568 INFO 26087 --- [ocp-schedules-29,1b9b49e94e69432b,b1674af2f655fb53] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:25:17.568 INFO 26087 --- [ocp-schedules-29,1b9b49e94e69432b,b1674af2f655fb53] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:25:18.261 INFO 26087 --- [ocp-schedules-30,b879e9bfd3cc9ca4,eb30d6ca71f61e11] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:25:18.262 INFO 26087 --- [ocp-schedules-30,b879e9bfd3cc9ca4,eb30d6ca71f61e11] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:25:18.355 INFO 26087 --- [prometheus-minute-schedule-1,6653c87a8316291c,3a55f6565c9ca3fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:18.641 WARN 26087 --- [ocp-async-2,e9e135159402fc8b,d46e6ac2479858e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:18.641 INFO 26087 --- [ocp-async-10,9129c8c39eee7b20,2f73e42d017e502d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:20.427 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:20.427 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:20.470 INFO 26087 --- [ocp-schedules-15,5d9e89aee23412eb,bf367aa32241c400] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:25:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:25:20.641 WARN 26087 --- [ocp-async-1,21576d983a582a95,6c5618484948a56c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:20.641 INFO 26087 --- [ocp-async-6,ac6fea409447744b,1e51aed6794655ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:21.355 INFO 26087 --- [prometheus-minute-schedule-1,9480593426b29112,007fd8ac68887ef9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:22.641 WARN 26087 --- [ocp-async-7,32cc5dcf7908dd76,7566690c634b424d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:22.641 INFO 26087 --- [ocp-async-3,2df8a3b13e33fa46,0209df4d17d89b08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:23.427 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:23.428 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:24.355 INFO 26087 --- [prometheus-minute-schedule-1,cab330196b9b4167,a795377f5e521167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:24.642 WARN 26087 --- [ocp-async-3,6a2f8508419cd4cd,5700ad317e1d4d30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:24.642 INFO 26087 --- [ocp-async-2,7740c07a7e0fa835,52e89d8012d1f674] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:26.428 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:26.429 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:26.558 INFO 26087 --- [ocp-async-8,8d15c738a8ea2143,c16ea25c17f7fcd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:25:26.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:25:26.624430+08:00, lastProcessTime=2025-04-08T21:25:16.624917+08:00 2025-04-08 21:25:26.642 WARN 26087 --- [ocp-async-10,c2a8dbd04df8d4ec,ad69fe67bc220625] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:26.642 INFO 26087 --- [ocp-async-6,393fb3e433df8591,685b2baf3bd3177b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:26.711 INFO 26087 --- [ocp-schedules-11,e0514180797e2803,91e2f38e2727b998] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:25:26.773 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:25:26.624430+08:00 2025-04-08 21:25:27.355 INFO 26087 --- [prometheus-minute-schedule-1,30ca747806835c88,e0d369413e5b7622] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:28.270 INFO 26087 --- [ocp-schedules-3,da78dac0a86c0c33,8caaa9af975858f0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:25:28.271 INFO 26087 --- [ocp-schedules-3,da78dac0a86c0c33,8caaa9af975858f0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:25:28.642 INFO 26087 --- [ocp-async-6,60b4ec325dabe985,00b558bd0efdc233] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:28.642 WARN 26087 --- [ocp-async-4,6694498fee13d606,811828432be019b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:29.429 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:29.430 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:30.355 INFO 26087 --- [prometheus-minute-schedule-1,7983c0c18e3e66a8,32ce337c8cddc074] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:30.495 INFO 26087 --- [ocp-schedules-4,a2679da5695d1e42,3db037eb63d48d8c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:25:30.642 INFO 26087 --- [ocp-async-9,6fb4503a94929296,e7b613d4d3cc236c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:30.642 WARN 26087 --- [ocp-async-2,c59d1b1bc6754fe2,b5921ac77cc3b085] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:32.430 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:32.431 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:32.642 WARN 26087 --- [ocp-async-1,cd55570c24c7353c,0af37fe80ced774e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:32.642 INFO 26087 --- [ocp-async-7,02230fcf0695d3a8,83652b27f4bd191c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:33.355 INFO 26087 --- [prometheus-minute-schedule-1,27ff8ab4acffa48e,233a6d2728278e62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:34.642 WARN 26087 --- [ocp-async-5,965a4257738ce26f,1c4187cd87f9cf1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:34.642 INFO 26087 --- [ocp-async-3,9d1395fc8d09cff2,db7c26d97f726421] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:34.978 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:34.979 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:25:34.979 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:25:34.980 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:25:34.980 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:25:34.982 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=2ms, period=120000ms 2025-04-08 21:25:35.431 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:35.431 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:36.355 INFO 26087 --- [prometheus-minute-schedule-1,322f5f6de36e3117,9be9e9d664b05f07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:36.558 INFO 26087 --- [ocp-async-3,5eef19dfebd80599,57d0e0464e39673a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:25:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:25:36.623208+08:00, lastProcessTime=2025-04-08T21:25:26.624430+08:00 2025-04-08 21:25:36.642 WARN 26087 --- [ocp-async-8,a358a4bf6df2e82a,45fff9892f236e97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:36.642 INFO 26087 --- [ocp-async-2,c34348a9a2c4a123,ee2daad1086de242] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:36.712 INFO 26087 --- [ocp-schedules-19,ea0ade7fb38d30d0,3e0fc880c324d6c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:25:36.728 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:25:36.623208+08:00 2025-04-08 21:25:37.392 INFO 26087 --- [ocp-schedules-24,7d417ef281105191,4fef7f454e0b49ba] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:25:37.428 INFO 26087 --- [ocp-schedules-24,7d417ef281105191,4fef7f454e0b49ba] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:25:37.428 INFO 26087 --- [ocp-schedules-24,7d417ef281105191,4fef7f454e0b49ba] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:25:38.277 INFO 26087 --- [ocp-schedules-22,854ca79a479d5348,a3a01bd3de9b5a46] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:25:38.278 INFO 26087 --- [ocp-schedules-22,854ca79a479d5348,a3a01bd3de9b5a46] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:25:38.432 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:38.432 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:38.643 WARN 26087 --- [ocp-async-10,5bf7df808f6a2d27,903866b3d4189428] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:38.643 INFO 26087 --- [ocp-async-7,2b9ee212edd45307,68e149adbd83a35b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:39.355 INFO 26087 --- [prometheus-minute-schedule-1,ed0f7266322090f4,ab05036b2ac5f4de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:40.002 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:25:40.002 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:25:40.511 INFO 26087 --- [ocp-schedules-28,051964f21c5be7b7,14ed194fecd34b7b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:25:40.643 INFO 26087 --- [ocp-async-5,5c58adb4c942cfe9,3919b87387d9b2f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:40.643 WARN 26087 --- [ocp-async-4,59f512be1920e7c9,b9b23f5cd88645b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:41.432 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:41.433 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:42.355 INFO 26087 --- [prometheus-minute-schedule-1,6631b56afcb30159,2a8a029eb861245a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:42.643 WARN 26087 --- [ocp-async-6,1eb628a91f5bb6bd,966c6b8b19781c90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:42.643 INFO 26087 --- [ocp-async-5,801ad9dd28761c87,68099275965a0086] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:44.433 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:44.434 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:44.643 WARN 26087 --- [ocp-async-10,a6e88d5a95543438,fbe4b5d66c844f40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:44.643 INFO 26087 --- [ocp-async-5,191ac7668bbba355,317b0baebf50c0fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:45.355 INFO 26087 --- [prometheus-minute-schedule-1,1e3bc042a7d94964,3485ce82a257a260] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:46.521 INFO 26087 --- [ocp-schedules-7,57a27b8fe01856d9,19c66c0ac887e5ec] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:25:46.521 INFO 26087 --- [ocp-schedules-1,de782c78489627c9,1b4b632742825593] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:25:46.521 INFO 26087 --- [ocp-schedules-1,de782c78489627c9,1b4b632742825593] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:25:46.523 INFO 26087 --- [ocp-schedules-25,40ed33727bb4c039,9ceae3c07cb17699] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:25:46.523 INFO 26087 --- [ocp-schedules-4,67a2134968988edc,4748ab9b670a5795] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:25:46.523 INFO 26087 --- [ocp-schedules-4,67a2134968988edc,4748ab9b670a5795] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:25:46.523 INFO 26087 --- [ocp-schedules-25,40ed33727bb4c039,9ceae3c07cb17699] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:25:46.524 INFO 26087 --- [ocp-schedules-24,bb2478d4eb12809f,c08ea8a69803e583] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:25:46.524 INFO 26087 --- [ocp-schedules-27,b516b967ae93d296,2ae968ec1927016b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:25:46.525 INFO 26087 --- [ocp-schedules-11,e16efce163e698c1,b4bc94a7b22f8585] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:25:46.531 INFO 26087 --- [ocp-schedules-19,bdb07419a27f8649,3c9dfdd2496449ce] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:25:46.558 INFO 26087 --- [ocp-async-5,1ffdc78178e8fd1f,2d4cc4a646bcb6fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:25:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:25:46.622870+08:00, lastProcessTime=2025-04-08T21:25:36.623208+08:00 2025-04-08 21:25:46.625 INFO 26087 --- [ocp-schedules-9,08f4a9794c9b843f,f8a4aabb86af12df] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:25:46.643 WARN 26087 --- [ocp-async-3,60ac726b7d45ca37,44c9b7e73bb7899f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:46.643 INFO 26087 --- [ocp-async-7,a0556be9748a729b,45fdba6477f89863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:46.713 INFO 26087 --- [ocp-schedules-26,f08ec4e052bc5a4c,49cc17a076cb828e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:25:46.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:25:46.622870+08:00 2025-04-08 21:25:47.434 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:47.434 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:48.283 INFO 26087 --- [ocp-schedules-18,201287603f9e4201,97820d34a931f646] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:25:48.284 INFO 26087 --- [ocp-schedules-18,201287603f9e4201,97820d34a931f646] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:25:48.355 INFO 26087 --- [prometheus-minute-schedule-1,71f238caf85fb049,774773279b69a7cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:48.643 WARN 26087 --- [ocp-async-9,c3951236650e11c4,a95733e4d144d883] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:48.643 INFO 26087 --- [ocp-async-7,02097bd1f0b6e515,af5a2e1e320a8262] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:50.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:50.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:50.530 INFO 26087 --- [ocp-schedules-20,1f4744c2ad0f4e73,7369a94ac0afe819] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:25:50.643 WARN 26087 --- [ocp-async-1,a2a3d7ba15c004c2,095f3c7985bab5e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:50.643 INFO 26087 --- [ocp-async-10,47111971fad49825,8919471e74ff6012] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:51.355 INFO 26087 --- [prometheus-minute-schedule-1,825c9e2d417e794d,9ab894ef53b7fe44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:52.644 WARN 26087 --- [ocp-async-4,3772ff0588a37296,33c05057116a3f53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:52.644 INFO 26087 --- [ocp-async-3,70dc4edf8fa8c753,fcbeb810024921af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:53.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:53.436 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:53.625 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,a08bbea8c888c0ab,432b303e46e5a499] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=a08bbea8c888c0ab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744118753621478,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 21:25:53.626 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-5,a08bbea8c888c0ab,432b303e46e5a499] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=a08bbea8c888c0ab, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744118753621478,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=4 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 21:25:54.355 INFO 26087 --- [prometheus-minute-schedule-1,039a8f9bb238b848,cd3ca478db0e18c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:54.644 WARN 26087 --- [ocp-async-9,ab0415f811ee5d4c,921e75f73e21c9a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:54.644 INFO 26087 --- [ocp-async-3,1b6a92addb9b4add,7b1cc60bc964061b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:56.436 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:56.436 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:25:56.559 INFO 26087 --- [ocp-async-3,765b91c57dafa271,522d176d2edaaf2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:25:56.574 INFO 26087 --- [ocp-schedules-9,06c62aee989ef52f,41e31538fe5a79bf] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:25:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:25:56.622900+08:00, lastProcessTime=2025-04-08T21:25:46.622870+08:00 2025-04-08 21:25:56.644 WARN 26087 --- [ocp-async-10,7a385c5a196315e4,ef2ae2350578656e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:56.644 INFO 26087 --- [ocp-async-1,3e679bc9cdf72749,14bd437433b3d3d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:56.714 INFO 26087 --- [ocp-schedules-4,ab584c044c3a15a1,b1a2b62291a43b5c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:25:56.756 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:25:56.622900+08:00 2025-04-08 21:25:57.355 INFO 26087 --- [prometheus-minute-schedule-1,ee52296b74996a8f,486413f05c3f62aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118760, interval=60 2025-04-08 21:25:57.429 INFO 26087 --- [ocp-schedules-15,e42a2fea5ab6900e,fa177eb5fe0f781f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:25:57.441 INFO 26087 --- [ocp-schedules-15,e42a2fea5ab6900e,fa177eb5fe0f781f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:25:57.441 INFO 26087 --- [ocp-schedules-15,e42a2fea5ab6900e,fa177eb5fe0f781f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:25:58.289 INFO 26087 --- [ocp-schedules-13,7791eb0b84d5fa5e,9869dd82eaa3efcb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:25:58.290 INFO 26087 --- [ocp-schedules-13,7791eb0b84d5fa5e,9869dd82eaa3efcb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:25:58.644 WARN 26087 --- [ocp-async-6,5d609eeb0eb8c7f7,b0e3115138fbb55a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:25:58.644 INFO 26087 --- [ocp-async-4,1b637abc82c79fde,c2758a6573eadf24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:25:59.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:25:59.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:00.355 INFO 26087 --- [prometheus-minute-schedule-1,5db4d0d93587fa89,ad4b860f3d24a17e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:00.547 INFO 26087 --- [ocp-schedules-7,1f46df6d2adf8523,ffb64fdebf50455a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:26:00.644 WARN 26087 --- [ocp-async-9,23c6a896111f919b,9bf1c1635000878c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:00.644 INFO 26087 --- [ocp-async-7,57c082b48fa27b5c,0af9f3a4504cbc95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:02.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:02.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:02.644 WARN 26087 --- [ocp-async-10,ccc2b4a0b01f3c02,1295a5ed53d530c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:02.644 INFO 26087 --- [ocp-async-2,f47eb8a3c3f0e986,34730486572ae298] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:03.355 INFO 26087 --- [prometheus-minute-schedule-1,0f62e4706d9a7712,e55f86a007833e2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:04.644 WARN 26087 --- [ocp-async-1,39513d31a69a34ec,049296725cc80692] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:04.644 INFO 26087 --- [ocp-async-6,45da42041ea2c4ae,b6d7b6ebbd2cf09e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:05.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:05.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:06.355 INFO 26087 --- [prometheus-minute-schedule-1,4364d34af52aaeb0,b8704b7db8256f63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:06.355 INFO 26087 --- [prometheus-cache-monitor1,c2a35e1edf53b400,558a61e59fa76ba5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:26:06.355 INFO 26087 --- [prometheus-cache-monitor1,c2a35e1edf53b400,558a61e59fa76ba5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:26:06.559 INFO 26087 --- [ocp-async-6,4918457eab6cf512,9548d76ce4f45f17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:26:06.559 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:22:06.495, end=2025-04-08T21:24:06.545} 2025-04-08 21:26:06.564 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:26:06.564 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:26:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:26:06.622946+08:00, lastProcessTime=2025-04-08T21:25:56.622900+08:00 2025-04-08 21:26:06.645 WARN 26087 --- [ocp-async-8,b5b8b4e6de7dfff0,7dbb9d657239f3f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:06.645 INFO 26087 --- [ocp-async-9,91efa4161a3206e7,63e8f4d33983d3bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:06.715 INFO 26087 --- [ocp-schedules-7,0905cd0a10ed6f5c,106169ba49ea76cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:26:06.732 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:26:06.622946+08:00 2025-04-08 21:26:08.296 INFO 26087 --- [ocp-schedules-6,88a6af79339be51a,d38107f7796c8a1d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:26:08.297 INFO 26087 --- [ocp-schedules-6,88a6af79339be51a,d38107f7796c8a1d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:26:08.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:08.439 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:08.645 WARN 26087 --- [ocp-async-10,d168527ae2f78a26,56aa978db8f2985b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:08.645 INFO 26087 --- [ocp-async-2,ab3212f8ad78d1f2,e3ca276a2c64602c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:09.355 INFO 26087 --- [prometheus-minute-schedule-1,33e989db2fd0a65b,307d000797d102c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:10.001 INFO 26087 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:26:10.001 INFO 26087 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:26:10.569 INFO 26087 --- [ocp-schedules-2,82c3c9d1d90d49cd,a56107824925de81] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:26:10.601 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:26:10.645 WARN 26087 --- [ocp-async-6,4eb2f6b20d982836,dbb9a8272a1b55d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:10.645 INFO 26087 --- [ocp-async-5,09a7f3ec58a2fadb,aac30a55ced17ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:11.439 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:11.452 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:12.355 INFO 26087 --- [prometheus-minute-schedule-1,44fe53b434fc4906,fcbfd6e4dbae83d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:12.458 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118700, nodeCount=75, affectRows=75 2025-04-08 21:26:12.458 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118700, nodeCount=75, validCount=75 2025-04-08 21:26:12.459 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118700, nodeCount=83, affectRows=83 2025-04-08 21:26:12.459 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118700, nodeCount=70, affectRows=70 2025-04-08 21:26:12.459 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118700, nodeCount=70, affectRows=70 2025-04-08 21:26:12.459 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118700, nodeCount=80, affectRows=80 2025-04-08 21:26:12.460 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118700, nodeCount=70, validCount=70 2025-04-08 21:26:12.460 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118700, nodeCount=69, validCount=69 2025-04-08 21:26:12.460 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118700, nodeCount=83, validCount=83 2025-04-08 21:26:12.460 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118700, nodeCount=80, validCount=80 2025-04-08 21:26:12.645 WARN 26087 --- [ocp-async-7,32ea4771e9044ca5,bf6546ad67076e02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:12.645 INFO 26087 --- [ocp-async-4,68c14ef562e309ad,7977e83c17e60e36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:14.452 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:14.452 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:14.645 WARN 26087 --- [ocp-async-3,9d495423af45d5f3,43cc83cfb83da7c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:14.645 INFO 26087 --- [ocp-async-9,a34ed9c49bef827f,bed7f584e1034ca9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:15.355 INFO 26087 --- [prometheus-minute-schedule-1,60423a3f231d7b15,ffa6d7cd007fc40b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:16.517 INFO 26087 --- [ocp-schedules-14,4142b1994c962086,a44772aa20f80d90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:26:16.517 INFO 26087 --- [ocp-schedules-1,06bf45fef913adc5,53a6093773f1ee66] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:26:16.521 INFO 26087 --- [ocp-schedules-13,b13f55bb0fb374c6,047d8d0edd1ece93] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:26:16.521 INFO 26087 --- [ocp-schedules-13,b13f55bb0fb374c6,047d8d0edd1ece93] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:26:16.521 INFO 26087 --- [ocp-schedules-26,aa7f93f695a018e2,c0490b75c64321fe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:26:16.523 INFO 26087 --- [ocp-schedules-14,4142b1994c962086,a44772aa20f80d90] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:26:16.523 INFO 26087 --- [ocp-schedules-4,a51f7c0edb28dd8f,fbe9d4dcdcf890ab] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:26:16.523 INFO 26087 --- [ocp-schedules-11,b1eb20e91c222a32,0b7cb9b49b676ab1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:26:16.523 INFO 26087 --- [ocp-schedules-4,a51f7c0edb28dd8f,fbe9d4dcdcf890ab] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:26:16.523 INFO 26087 --- [ocp-schedules-11,b1eb20e91c222a32,0b7cb9b49b676ab1] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:26:16.524 INFO 26087 --- [ocp-schedules-6,2de6fd4f2891459c,6672639904a7bd82] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:26:16.524 INFO 26087 --- [ocp-schedules-15,3ea60ce05c87cd9d,ab509f7486d9e97a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:26:16.524 INFO 26087 --- [ocp-schedules-3,9917e6303307e3a8,915de3fd0a4da667] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:26:16.525 INFO 26087 --- [ocp-schedules-1,06bf45fef913adc5,53a6093773f1ee66] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:26:16.559 INFO 26087 --- [ocp-async-9,35d049e485ba8c5a,3d52eaca5d169e37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:26:16.563 INFO 26087 --- [ocp-schedules-9,3a6f0e0968eb8e67,552b802bdc48efc4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:26:16.601 INFO 26087 --- [ocp-schedules-8,d354dc457f3ab921,c639915f52ea12f7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:26:16.610 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:26:16.610 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:26:16.613 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:26:16.613 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:26:16.613 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:26:16.614 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:26:16.614 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:26:16.614 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:26:16.616 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:26:16.616 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:26:16.616 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:26:16.619 INFO 26087 --- [ocp-schedules-7,887b8c3ba6827c52,812cabe14287a045] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:26:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:26:16.622670+08:00, lastProcessTime=2025-04-08T21:26:06.622946+08:00 2025-04-08 21:26:16.627 INFO 26087 --- [ocp-schedules-19,88385e8218e75bd8,4abb54a3698e900c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:26:16.637 INFO 26087 --- [ocp-schedules-21,499b1e692f3bc78d,b4e3bf52403eb820] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:26:16.637 INFO 26087 --- [ocp-schedules-21,499b1e692f3bc78d,b4e3bf52403eb820] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:26:16.645 INFO 26087 --- [ocp-async-5,a818c0c0a571beac,6c840ac31a14adc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:16.645 WARN 26087 --- [ocp-async-7,ff54412bdba3381c,4ad9c332b4de6b78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:16.651 INFO 26087 --- [ocp-schedules-29,7a239a21c90823b1,b2fafde56b72c117] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:26:16.651+08:00, clusters=[] 2025-04-08 21:26:16.651 INFO 26087 --- [ocp-schedules-26,69785331191d6698,24ae8f2a172020cf] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:26:16.651 INFO 26087 --- [ocp-schedules-26,69785331191d6698,24ae8f2a172020cf] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:26:16.652 INFO 26087 --- [ocp-schedules-11,091ea53edbf8197b,593f6604d8b040da] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:26:16.656 INFO 26087 --- [ocp-schedules-15,471d7b13199be599,568f1f9e37c2cf09] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:26:16.656+08:00, clusters=[] 2025-04-08 21:26:16.676 INFO 26087 --- [ocp-schedules-6,d4e40f4c0d3eb668,c0aaeb86dd6cf1ba] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:26:16.676 INFO 26087 --- [ocp-schedules-6,d4e40f4c0d3eb668,c0aaeb86dd6cf1ba] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:26:16.678 INFO 26087 --- [ocp-schedules-6,d4e40f4c0d3eb668,c0aaeb86dd6cf1ba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:26:16.678 INFO 26087 --- [ocp-schedules-6,d4e40f4c0d3eb668,c0aaeb86dd6cf1ba] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:26:16.682 INFO 26087 --- [ocp-schedules-1,d66e638eaf4d04ab,8edefaccadc0ef1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:26:16.683 INFO 26087 --- [ocp-schedules-1,d66e638eaf4d04ab,8edefaccadc0ef1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:26:16.683 INFO 26087 --- [ocp-schedules-1,d66e638eaf4d04ab,8edefaccadc0ef1e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:26:16.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:26:16.622670+08:00 2025-04-08 21:26:16.716 INFO 26087 --- [ocp-schedules-28,cfd9a8059687a651,8e1151b083c2b1af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:26:16.820 INFO 26087 --- [ocp-schedules-20,6f7d58fdd8a20c5c,e0c8754edf9cacd5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:26:16.832 INFO 26087 --- [ocp-schedules-22,90dea29ae43d5b59,a9030d593779027c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:26:16.846 INFO 26087 --- [ocp-schedules-9,2c17558a4801940a,8e3e0c6956c32fa3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:26:16.867 INFO 26087 --- [ocp-schedules-12,8b99a22f576bc5f7,6a274e8fd427aaf7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:26:16.867 INFO 26087 --- [ocp-schedules-12,8b99a22f576bc5f7,6a274e8fd427aaf7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:26:16.867 INFO 26087 --- [ocp-schedules-12,8b99a22f576bc5f7,6a274e8fd427aaf7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:26:16.867 INFO 26087 --- [ocp-schedules-12,8b99a22f576bc5f7,6a274e8fd427aaf7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:26:16.867 INFO 26087 --- [ocp-schedules-12,8b99a22f576bc5f7,6a274e8fd427aaf7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:26:16.867 INFO 26087 --- [ocp-schedules-12,8b99a22f576bc5f7,6a274e8fd427aaf7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:26:17.442 INFO 26087 --- [ocp-schedules-7,583158dab2d34adf,0a16c3777a975ce0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:26:17.453 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:17.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:17.467 INFO 26087 --- [ocp-schedules-7,583158dab2d34adf,0a16c3777a975ce0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:26:17.467 INFO 26087 --- [ocp-schedules-7,583158dab2d34adf,0a16c3777a975ce0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:26:17.571 INFO 26087 --- [ocp-schedules-13,50ac46295d6f3d7b,ccb7f310ec22c692] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:26:17.571 INFO 26087 --- [ocp-schedules-13,50ac46295d6f3d7b,ccb7f310ec22c692] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:26:18.300 INFO 26087 --- [ocp-schedules-23,9eb4cb0d6ce0baac,aebde074bb88b527] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:26:18.302 INFO 26087 --- [ocp-schedules-23,9eb4cb0d6ce0baac,aebde074bb88b527] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:26:18.355 INFO 26087 --- [prometheus-minute-schedule-1,99b74300390b9d03,e8fc0d02f2d3b5b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:18.645 WARN 26087 --- [ocp-async-3,f2c64f954a37a949,512e8d1979c9a33b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:18.645 INFO 26087 --- [ocp-async-2,29dd7cc647f6d2f8,9728341701a4c4ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:20.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:20.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:20.586 INFO 26087 --- [ocp-schedules-23,a29b1f8698f25356,72bb0279d3dbfef7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:26:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:26:20.646 INFO 26087 --- [ocp-async-6,c058d1a6c6ec9bdd,ad6ca34c71c270b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:20.646 WARN 26087 --- [ocp-async-9,658581dc5a2fcee9,8742584f3f1d6671] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:21.355 INFO 26087 --- [prometheus-minute-schedule-1,546b6f49b0997e0a,c2171a11212028ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:22.646 WARN 26087 --- [ocp-async-7,19e89c9c277ea8af,79bb76162d02581b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:22.646 INFO 26087 --- [ocp-async-8,5dddbf13de8098fc,674cf7a08b9cf6d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:23.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:23.455 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:24.355 INFO 26087 --- [prometheus-minute-schedule-1,8e2e8210285de3ab,f935222121a29346] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:24.646 WARN 26087 --- [ocp-async-3,33c93db657b9bb86,84903809055358bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:24.646 INFO 26087 --- [ocp-async-8,effb6f430500b703,3872bef452111911] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:26.455 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:26.456 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:26.559 INFO 26087 --- [ocp-async-2,548333d8c07191a1,f3d47b0d9d140e30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:26:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:26:26.623743+08:00, lastProcessTime=2025-04-08T21:26:16.622670+08:00 2025-04-08 21:26:26.646 WARN 26087 --- [ocp-async-6,563df89bb66ea765,147a3b829ee0d4f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:26.646 INFO 26087 --- [ocp-async-7,4da0d7666ddcf7b4,42c92e4d6fc33db7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:26.709 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:26:26.623743+08:00 2025-04-08 21:26:26.717 INFO 26087 --- [ocp-schedules-3,14f09a4a600a2d73,666a4cd5ebc7bdad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:26:27.355 INFO 26087 --- [prometheus-minute-schedule-1,ce9075530e70a8c3,b66b01de05653652] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:28.307 INFO 26087 --- [ocp-schedules-2,236fc60941efb5e1,e4a56c534c537b48] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:26:28.308 INFO 26087 --- [ocp-schedules-2,236fc60941efb5e1,e4a56c534c537b48] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:26:28.646 WARN 26087 --- [ocp-async-1,5398ae573a8ee490,08594a5085d2353a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:28.646 INFO 26087 --- [ocp-async-5,b839c71199f60d27,054fa4340df443e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:29.456 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:29.456 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:30.355 INFO 26087 --- [prometheus-minute-schedule-1,5fc82696508bae4d,dedb1e17cac5805b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:30.614 INFO 26087 --- [ocp-schedules-5,fb9105a8208ef168,640fcfe3746d7008] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:26:30.646 WARN 26087 --- [ocp-async-2,0b2685a52c1fc996,7c8c1b5f9bb2072b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:30.646 INFO 26087 --- [ocp-async-8,ea027e274f79f4c3,dd0729d4a9a629c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:32.457 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:32.457 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:32.646 WARN 26087 --- [ocp-async-9,d84736852d242d39,054112fb32a57930] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:32.646 INFO 26087 --- [ocp-async-10,34bd9f9ca60d5771,1241695b257a12ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:33.355 INFO 26087 --- [prometheus-minute-schedule-1,80d024f2492b200a,929b3512d8b7efa8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:34.647 WARN 26087 --- [ocp-async-1,95f13529c595038e,1954f9a1b959aa42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:34.647 INFO 26087 --- [ocp-async-7,1c5f9fb0394ea2f4,632fd239591c4c39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:35.457 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:35.458 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:36.355 INFO 26087 --- [prometheus-minute-schedule-1,16cae290c1944195,edc94c84c20d49be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:36.559 INFO 26087 --- [ocp-async-7,347073f4ec8ff88a,704ccf35e749bc11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:26:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:26:36.622741+08:00, lastProcessTime=2025-04-08T21:26:26.623743+08:00 2025-04-08 21:26:36.647 WARN 26087 --- [ocp-async-8,0de9385c0c503568,9ee060f08a92953b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:36.647 INFO 26087 --- [ocp-async-4,2af02e37aad26d8c,2d717a1bac60d10a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:36.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:26:36.622741+08:00 2025-04-08 21:26:36.719 INFO 26087 --- [ocp-schedules-20,cbbae793f227cb20,cf6440b40b6df17d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:26:37.468 INFO 26087 --- [ocp-schedules-30,0049064acedc3d59,fce3c1b726257717] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:26:37.477 INFO 26087 --- [ocp-schedules-30,0049064acedc3d59,fce3c1b726257717] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:26:37.477 INFO 26087 --- [ocp-schedules-30,0049064acedc3d59,fce3c1b726257717] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:26:38.317 INFO 26087 --- [ocp-schedules-26,a82801d97d062c04,747b2136c685f6af] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:26:38.318 INFO 26087 --- [ocp-schedules-26,a82801d97d062c04,747b2136c685f6af] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:26:38.458 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:38.459 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:38.647 INFO 26087 --- [ocp-async-9,af342908a78a127e,c5ab9cb740985574] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:38.647 WARN 26087 --- [ocp-async-6,d5d7ddfc7baa2c13,32688139b4caebef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:39.355 INFO 26087 --- [prometheus-minute-schedule-1,82f24fb83973abea,f854c251355ca635] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:40.001 INFO 26087 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:26:40.001 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:26:40.632 INFO 26087 --- [ocp-schedules-1,104f5c5691cf3027,a77531822c90d879] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:26:40.647 INFO 26087 --- [ocp-async-1,aba7ec5bc96a239b,4fea1ae94a6e3c9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:40.647 WARN 26087 --- [ocp-async-3,0b9496125214fead,671ffe55ffde0bc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:41.459 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:41.459 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:42.355 INFO 26087 --- [prometheus-minute-schedule-1,cc42f39dbd8857f9,e768225002945cac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:42.647 INFO 26087 --- [ocp-async-1,47b4ad1163c5b52e,d6379bae71345b45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:42.647 WARN 26087 --- [ocp-async-8,0672b6da11ab496c,b626d877b6694c04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:44.459 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:44.460 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:44.647 INFO 26087 --- [ocp-async-1,d83319e6fa00fb45,a0b4d9b275112442] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:44.647 WARN 26087 --- [ocp-async-4,ce11427e83ea98a2,f6a81feed7e92a24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:45.355 INFO 26087 --- [prometheus-minute-schedule-1,51b904beea157383,1c67a97bde6cce82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:46.521 INFO 26087 --- [ocp-schedules-5,0d6b76230bc17b4b,b52b2b571f34932e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:26:46.521 INFO 26087 --- [ocp-schedules-14,435f701a4c9fbce3,19f07439c54e09df] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:26:46.521 INFO 26087 --- [ocp-schedules-14,435f701a4c9fbce3,19f07439c54e09df] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:26:46.523 INFO 26087 --- [ocp-schedules-19,8dc5738722c45681,75d53576d3134869] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:26:46.523 INFO 26087 --- [ocp-schedules-19,8dc5738722c45681,75d53576d3134869] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:26:46.523 INFO 26087 --- [ocp-schedules-27,537c8bb80817b6b8,950b88886034508d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:26:46.523 INFO 26087 --- [ocp-schedules-27,537c8bb80817b6b8,950b88886034508d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:26:46.524 INFO 26087 --- [ocp-schedules-29,febd2473ef3ddd17,b07faeed6969a03f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:26:46.524 INFO 26087 --- [ocp-schedules-17,90182a7965d304d9,f0ab1da8860b0747] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:26:46.524 INFO 26087 --- [ocp-schedules-25,1b3ff55a3dd302f0,00f3f82d3fa9acba] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:26:46.526 INFO 26087 --- [ocp-schedules-21,49bee6e5f6db48cf,4dd94f954a8ab2a4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:26:46.559 INFO 26087 --- [ocp-async-1,148487c7de9f59d1,adb7b894b4ffc6f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:26:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:26:46.623174+08:00, lastProcessTime=2025-04-08T21:26:36.622741+08:00 2025-04-08 21:26:46.628 INFO 26087 --- [ocp-schedules-2,77a1b24e0916ee53,b6759f60ad09a549] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:26:46.647 WARN 26087 --- [ocp-async-7,79d331e59aa361b9,1828a8bbc88ed4ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:46.647 INFO 26087 --- [ocp-async-9,6c58c9649ab64bdc,ed7fabd0cd243adc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:46.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:26:46.623174+08:00 2025-04-08 21:26:46.720 INFO 26087 --- [ocp-schedules-23,73493184f1661bf7,5970e60bc89a38c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:26:47.460 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:47.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:48.324 INFO 26087 --- [ocp-schedules-22,963cebe9142db094,ff51a6abbeddabd4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:26:48.325 INFO 26087 --- [ocp-schedules-22,963cebe9142db094,ff51a6abbeddabd4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:26:48.355 INFO 26087 --- [prometheus-minute-schedule-1,931769d50bd0909b,047b427794d02f58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:48.648 INFO 26087 --- [ocp-async-9,5827e7058566702d,92da8ad981d92d57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:48.648 WARN 26087 --- [ocp-async-8,a8eb76f8ded9bfbc,14cc209dc3038d93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:50.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:50.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:50.648 WARN 26087 --- [ocp-async-6,d6abdc7c15c7c2ba,dadfe65ee2babf70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:50.648 INFO 26087 --- [ocp-async-9,43c7d803e7af147c,7d168e17988a1213] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:50.664 INFO 26087 --- [ocp-schedules-30,6438b6cef926b8fa,fdcc2c750595ca3c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:26:51.355 INFO 26087 --- [prometheus-minute-schedule-1,d5722b42864919e9,1107754483ec7cfd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:52.648 WARN 26087 --- [ocp-async-3,2f4770aa0b9b628e,d0c77ddb9747c21d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:52.648 INFO 26087 --- [ocp-async-9,4161144b9cf52ebf,68a07c1bb43f0b62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:53.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:53.462 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:54.355 INFO 26087 --- [prometheus-minute-schedule-1,6cae7d44c3f97211,72168ebe2de9c080] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:54.648 WARN 26087 --- [ocp-async-9,d804b869db406831,115014d5314eb751] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:54.648 INFO 26087 --- [ocp-async-8,0c74041fca9357d7,20aaecd901535ed4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:56.462 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:56.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:26:56.559 INFO 26087 --- [ocp-async-4,a247cde134540cd6,af1f9857b304b1cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:26:56.575 INFO 26087 --- [ocp-schedules-15,580a747b636557e0,61d2d771171af442] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:26:56.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:26:56.624718+08:00, lastProcessTime=2025-04-08T21:26:46.623174+08:00 2025-04-08 21:26:56.648 WARN 26087 --- [ocp-async-6,4e61287eb583bdb8,adf581ee4fcba598] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:56.648 INFO 26087 --- [ocp-async-1,fc7f8e26df58f445,e267f6bcc189a123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:56.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:26:56.624718+08:00 2025-04-08 21:26:56.722 INFO 26087 --- [ocp-schedules-26,c0034ba5c245ab63,f941d00241825afa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:26:57.355 INFO 26087 --- [prometheus-minute-schedule-1,f284ffaff6e5aad8,6b6f4eafcfeb7349] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118820, interval=60 2025-04-08 21:26:57.479 INFO 26087 --- [ocp-schedules-11,0311cf207abdafee,ad1399a59ed3b2d6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:26:57.482 INFO 26087 --- [ocp-schedules-11,0311cf207abdafee,ad1399a59ed3b2d6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:26:57.482 INFO 26087 --- [ocp-schedules-11,0311cf207abdafee,ad1399a59ed3b2d6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:26:58.331 INFO 26087 --- [ocp-schedules-23,0568d9387ac64bad,946fd4cc325e4bd2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:26:58.332 INFO 26087 --- [ocp-schedules-23,0568d9387ac64bad,946fd4cc325e4bd2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:26:58.648 WARN 26087 --- [ocp-async-5,3319d1b8dc7c965c,332a2ab31abca0a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:26:58.648 INFO 26087 --- [ocp-async-7,ef92c3f8d0f4b050,8a438861c7773b3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:26:59.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:26:59.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:00.355 INFO 26087 --- [prometheus-minute-schedule-1,9b3a3e7d9d6fdd39,b8d001961e55a2f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:00.648 WARN 26087 --- [ocp-async-2,9d08a66fbb3bb827,a9074a9c064f7bf0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:00.648 INFO 26087 --- [ocp-async-8,eab76df515f8efc2,9ab1dee87aa365e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:00.681 INFO 26087 --- [ocp-schedules-10,6687471d655da98b,223b6c25c3007780] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:27:02.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:02.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:02.649 WARN 26087 --- [ocp-async-4,5121ed2ffcf8f9a1,91dc2475106906c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:02.649 INFO 26087 --- [ocp-async-6,ed433c1b628b40b6,b7ad1f266a10f967] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:03.355 INFO 26087 --- [prometheus-minute-schedule-1,279e90063e72f50e,26796a1e25077e9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:04.649 WARN 26087 --- [ocp-async-9,21b56b1b932037bf,1cbf5a536a372140] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:04.649 INFO 26087 --- [ocp-async-6,2990a9a23e9a53fe,0484cca405ce1554] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:05.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:05.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:06.355 INFO 26087 --- [prometheus-minute-schedule-1,43c6a379e62852dd,ca4d36492d0fac27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:06.355 INFO 26087 --- [prometheus-cache-monitor1,557cd73841eaa19a,790a3181a47c0b94] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:27:06.355 INFO 26087 --- [prometheus-cache-monitor1,557cd73841eaa19a,790a3181a47c0b94] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:27:06.560 INFO 26087 --- [ocp-async-6,f20b909b083cfb3b,4cf5ebf6fdc2dfb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:27:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:27:06.622978+08:00, lastProcessTime=2025-04-08T21:26:56.624718+08:00 2025-04-08 21:27:06.649 WARN 26087 --- [ocp-async-2,9cffbb70d9f49e67,ed973b261b1d40cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:06.649 INFO 26087 --- [ocp-async-7,19aa5ba542ab9120,f47a2503b295de6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:06.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:27:06.622978+08:00 2025-04-08 21:27:06.723 INFO 26087 --- [ocp-schedules-12,e49dfb613e969ea6,2a144f60126ba1e9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:27:08.337 INFO 26087 --- [ocp-schedules-7,bee9b761c61a804b,af14256c2e75e336] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:27:08.338 INFO 26087 --- [ocp-schedules-7,bee9b761c61a804b,af14256c2e75e336] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:27:08.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:08.465 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:08.649 WARN 26087 --- [ocp-async-10,c681a6a998ecd43a,c746ae4ad0b39e66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:08.649 INFO 26087 --- [ocp-async-3,012cba82f81e47aa,641eda40f79fa101] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:09.355 INFO 26087 --- [prometheus-minute-schedule-1,be31924bc43493a1,8a9423d4dde72e20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:10.001 INFO 26087 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:27:10.001 INFO 26087 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:27:10.601 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:27:10.649 WARN 26087 --- [ocp-async-1,dc6b817d9b2eca78,bcfabed6ec63305a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:10.649 INFO 26087 --- [ocp-async-9,726d5bbde5952c43,1497b1c6acc81d33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:10.704 INFO 26087 --- [ocp-schedules-11,03505623274e53bf,eed8ab20534c2d9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:27:11.465 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:11.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:12.355 INFO 26087 --- [prometheus-minute-schedule-1,dcdccb5ae937c6ff,a9601e6a930ee684] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:12.455 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118760, nodeCount=170, affectRows=170 2025-04-08 21:27:12.456 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118760, nodeCount=170, validCount=170 2025-04-08 21:27:12.459 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118760, nodeCount=208, affectRows=208 2025-04-08 21:27:12.461 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118760, nodeCount=207, validCount=207 2025-04-08 21:27:12.649 WARN 26087 --- [ocp-async-2,788d11211e564c1c,e76692d5ceb903a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:12.649 INFO 26087 --- [ocp-async-8,b0aa88e9a0790785,b41ff282927ab377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:14.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:14.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:14.649 WARN 26087 --- [ocp-async-8,b60f4c519790c39b,bab681cf14fc08a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:14.649 INFO 26087 --- [ocp-async-4,50cd4caaaa68260c,5fdbb32662d04aa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:15.355 INFO 26087 --- [prometheus-minute-schedule-1,2017e490a2887018,4be6149757b7883a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:16.517 INFO 26087 --- [ocp-schedules-28,534ac41200342a06,eeb08dfcaddebbf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:27:16.517 INFO 26087 --- [ocp-schedules-15,dad3096d85a83102,a31dc9206f824d82] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:27:16.521 INFO 26087 --- [ocp-schedules-2,e1bd4204ceb9f829,9cf518fe6b47cf56] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:27:16.521 INFO 26087 --- [ocp-schedules-2,e1bd4204ceb9f829,9cf518fe6b47cf56] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:27:16.521 INFO 26087 --- [ocp-schedules-8,0b66221d8046e9fa,d63dac95a3b8dd9f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:27:16.522 INFO 26087 --- [ocp-schedules-28,534ac41200342a06,eeb08dfcaddebbf5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:27:16.523 INFO 26087 --- [ocp-schedules-6,dbe3cab26fa7508c,a52da36c6da19f1e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:27:16.523 INFO 26087 --- [ocp-schedules-6,dbe3cab26fa7508c,a52da36c6da19f1e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:27:16.523 INFO 26087 --- [ocp-schedules-12,2a575069d9fe70ba,80b5e48d725cf1eb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:27:16.524 INFO 26087 --- [ocp-schedules-12,2a575069d9fe70ba,80b5e48d725cf1eb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:27:16.524 INFO 26087 --- [ocp-schedules-23,3eb1b02f223c29eb,dcb20854bdb92a9f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:27:16.524 INFO 26087 --- [ocp-schedules-20,79b7eba39fc528a6,06a04c1303151924] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:27:16.524 INFO 26087 --- [ocp-schedules-29,42cf0a434ec51140,568c3a61badf0b37] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:27:16.525 INFO 26087 --- [ocp-schedules-15,dad3096d85a83102,a31dc9206f824d82] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:27:16.536 INFO 26087 --- [ocp-schedules-9,d55d3deaaa9f87c9,144470faeede59b4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:27:16.560 INFO 26087 --- [ocp-async-1,b0fae15f9304f903,1b54a3ad26f5a506] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:27:16.586 INFO 26087 --- [ocp-schedules-4,f7ec81c12c535dc8,79dae03f510a6b38] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:27:16.586 INFO 26087 --- [ocp-schedules-4,f7ec81c12c535dc8,79dae03f510a6b38] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:27:16.602 INFO 26087 --- [ocp-schedules-4,e1e1fd3c484792c4,ecf4960e5c3eacbf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:27:16.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:27:16.624874+08:00, lastProcessTime=2025-04-08T21:27:06.622978+08:00 2025-04-08 21:27:16.629 INFO 26087 --- [ocp-schedules-5,ffce72807010b7cd,510bffc03d04abad] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:27:16.639 INFO 26087 --- [ocp-schedules-3,ebbb24a8464c4ed6,72ebc16a7234e2fa] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:27:16.639 INFO 26087 --- [ocp-schedules-3,ebbb24a8464c4ed6,72ebc16a7234e2fa] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:27:16.650 WARN 26087 --- [ocp-async-9,dc683cb36d83aec7,4243c0430dfc32c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:16.650 INFO 26087 --- [ocp-async-2,640ad21abdd5bcb7,809609b0c25c0802] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:16.652 INFO 26087 --- [ocp-schedules-8,28b83c0600e326d0,9f78e49ae074b225] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:27:16.652+08:00, clusters=[] 2025-04-08 21:27:16.653 INFO 26087 --- [ocp-schedules-12,2fcbeda730c877b6,b16f12cb7a796847] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:27:16.653 INFO 26087 --- [ocp-schedules-12,2fcbeda730c877b6,b16f12cb7a796847] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:27:16.653 INFO 26087 --- [ocp-schedules-6,fea8ce72ec48f66b,7e2032edea893bb8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:27:16.657 INFO 26087 --- [ocp-schedules-15,4f94364379b181d4,092d5dbb8e15a2ea] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:27:16.657+08:00, clusters=[] 2025-04-08 21:27:16.678 INFO 26087 --- [ocp-schedules-23,c780fc78bb805b16,cce456b8c5622fff] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:27:16.678 INFO 26087 --- [ocp-schedules-23,c780fc78bb805b16,cce456b8c5622fff] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:27:16.679 INFO 26087 --- [ocp-schedules-23,c780fc78bb805b16,cce456b8c5622fff] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:27:16.679 INFO 26087 --- [ocp-schedules-23,c780fc78bb805b16,cce456b8c5622fff] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:27:16.683 INFO 26087 --- [ocp-schedules-20,afdb0fd472736368,51ecbd6c1fd79326] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:27:16.684 INFO 26087 --- [ocp-schedules-20,afdb0fd472736368,51ecbd6c1fd79326] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:27:16.684 INFO 26087 --- [ocp-schedules-20,afdb0fd472736368,51ecbd6c1fd79326] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:27:16.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:27:16.624874+08:00 2025-04-08 21:27:16.724 INFO 26087 --- [ocp-schedules-30,d1ce38b4d722fced,282a8449c95f56c9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:27:16.824 INFO 26087 --- [ocp-schedules-10,46d4059585fd7431,3e252125e6eb81cc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:27:16.833 INFO 26087 --- [ocp-schedules-13,9e470d153bee3add,c8716551cd75c502] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:27:16.833 INFO 26087 --- [ocp-schedules-26,682c3496c4ebbac2,d0e685293879660c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:27:16.847 INFO 26087 --- [ocp-schedules-25,e4a11b03aeea3eea,03880e7ce837af9e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:27:16.868 INFO 26087 --- [ocp-schedules-21,e46ecdeb2c90996f,38eb7fd09a426280] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:27:16.868 INFO 26087 --- [ocp-schedules-21,e46ecdeb2c90996f,38eb7fd09a426280] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:27:16.868 INFO 26087 --- [ocp-schedules-21,e46ecdeb2c90996f,38eb7fd09a426280] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:27:16.868 INFO 26087 --- [ocp-schedules-21,e46ecdeb2c90996f,38eb7fd09a426280] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:27:16.869 INFO 26087 --- [ocp-schedules-21,e46ecdeb2c90996f,38eb7fd09a426280] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:27:16.869 INFO 26087 --- [ocp-schedules-21,e46ecdeb2c90996f,38eb7fd09a426280] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:27:17.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:17.467 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:17.483 INFO 26087 --- [ocp-schedules-3,011f3d002e98a75c,e8c8eaccd315bbed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:27:17.491 INFO 26087 --- [ocp-schedules-3,011f3d002e98a75c,e8c8eaccd315bbed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:27:17.491 INFO 26087 --- [ocp-schedules-3,011f3d002e98a75c,e8c8eaccd315bbed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:27:17.574 INFO 26087 --- [ocp-schedules-17,a746fda61edb79e4,e32acb4d037a5bbb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:27:17.574 INFO 26087 --- [ocp-schedules-17,a746fda61edb79e4,e32acb4d037a5bbb] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:27:18.344 INFO 26087 --- [ocp-schedules-30,cb60461954eadbab,e3e834ff464f1eab] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:27:18.345 INFO 26087 --- [ocp-schedules-30,cb60461954eadbab,e3e834ff464f1eab] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:27:18.355 INFO 26087 --- [prometheus-minute-schedule-1,c0f332288b4b7612,2d5b696375b8415c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:18.650 WARN 26087 --- [ocp-async-2,dfdf6551faf9c0c9,053ca5380852e2d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:18.650 INFO 26087 --- [ocp-async-8,a46f3b6fe3b821e8,8b024fee751e3a5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:20.467 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:20.468 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:27:20.650 WARN 26087 --- [ocp-async-8,edb2ec67d59ca529,5f394a27c3745569] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:20.650 INFO 26087 --- [ocp-async-3,0c1e1a1294913049,12b8e9160f762834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:20.720 INFO 26087 --- [ocp-schedules-21,3a345a26d1707f56,2439803a6a05e68c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:27:21.355 INFO 26087 --- [prometheus-minute-schedule-1,8377881be1b45776,738664cef3902f55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:22.650 WARN 26087 --- [ocp-async-1,412b5b4ad18a3843,5719aafea4a2b522] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:22.650 INFO 26087 --- [ocp-async-9,b03f84ea9d1b597e,92046fc4512a7dc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:23.468 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:23.468 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:24.355 INFO 26087 --- [prometheus-minute-schedule-1,d0b633ec99fed8ca,316325a67a134ece] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:24.650 WARN 26087 --- [ocp-async-2,ce8ba804c5130d7b,012c3413f1614b10] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:24.650 INFO 26087 --- [ocp-async-4,e94096aa288adf69,eb9679b1990b19bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:26.386 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:27:26.386 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:27:26.389 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:27:26.389 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:27:26.389 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:27:26.389 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:27:26.392 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:27:26.392 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:27:26.392 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:27:26.392 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:27:26.413 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:27:26.418 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:27:26.418 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:27:26.419 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:27:26.419 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:27:26.419 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:27:26.424 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:27:26.468 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:26.469 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:26.560 INFO 26087 --- [ocp-async-10,a91552bb13ef405d,646f6c9e141436dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:27:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:27:26.624250+08:00, lastProcessTime=2025-04-08T21:27:16.624874+08:00 2025-04-08 21:27:26.650 WARN 26087 --- [ocp-async-5,46570c628a0e6194,861539084c152b13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:26.650 INFO 26087 --- [ocp-async-3,f819d3a2d513dc34,ec2004a62dffc4be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:26.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:27:26.624250+08:00 2025-04-08 21:27:26.725 INFO 26087 --- [ocp-schedules-13,ad689d01173660e0,63e4b5b7c194085a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:27:26.725 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:27:27.040 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.647784371S 2025-04-08 21:27:27.040 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:27:27.098 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:27:27.098 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:27:27.120 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:27:27.154 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:27:27.154 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:27:27.154 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:27:27.154 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:27:27.154 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:27:27.155 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:27:27.161 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:27:27.273 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] o.s.boot.SpringApplication : Started application in 0.884 seconds (JVM running for 1552.646) 2025-04-08 21:27:27.273 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.88453241S 2025-04-08 21:27:27.355 INFO 26087 --- [prometheus-minute-schedule-1,fde139974bf98eb5,b2df679214934e0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:27.526 INFO 26087 --- [ocp-schedules-27,98c677e6a57975fb,fdfb014216ac2c1a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:27:28.350 INFO 26087 --- [ocp-schedules-19,cad09b9f007c9513,5689a9e181d99ce0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:27:28.351 INFO 26087 --- [ocp-schedules-19,cad09b9f007c9513,5689a9e181d99ce0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:27:28.650 WARN 26087 --- [ocp-async-2,cfa9d004b78e5d83,933d1be4277e622f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:28.650 INFO 26087 --- [ocp-async-7,7d868983a75b7d4f,a658c34064e659c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:29.469 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:29.470 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:30.355 INFO 26087 --- [prometheus-minute-schedule-1,4ed6613e6b983fa1,009c2ea9dc76de96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:30.651 WARN 26087 --- [ocp-async-9,b0a73d1ae3da090f,48cf76061d09fbfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:30.651 INFO 26087 --- [ocp-async-4,0181bdf9662b74ca,57a7b7a881e6ff56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:30.749 INFO 26087 --- [ocp-schedules-20,50adce518d206795,49b33751c932e943] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:27:32.470 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:32.470 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:32.651 WARN 26087 --- [ocp-async-5,b1d38ffc7a9347c6,cec8238d3d3d3ddc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:32.651 INFO 26087 --- [ocp-async-4,fa423e01994df3f0,23eb3463e1b2ae63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:33.355 INFO 26087 --- [prometheus-minute-schedule-1,66808b73b0d654af,30d620e6543cbbb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:34.651 WARN 26087 --- [ocp-async-3,75c5aa4bd8dd61c7,0c9dab1c003a86e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:34.651 INFO 26087 --- [ocp-async-6,68f8795a6aef5aa9,0b112b23191c5ad0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:34.992 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:27:34.993 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:27:34.993 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:27:34.994 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:27:34.994 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:27:34.995 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:27:35.470 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:35.471 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:36.355 INFO 26087 --- [prometheus-minute-schedule-1,5cbc3d02df8b5d33,6fdd2d2a0bb0e11e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:36.560 INFO 26087 --- [ocp-async-6,1c7e589ff9bd43a8,fde8aa8774623efa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:27:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:27:36.622674+08:00, lastProcessTime=2025-04-08T21:27:26.624250+08:00 2025-04-08 21:27:36.651 WARN 26087 --- [ocp-async-9,32bdbd36b533db87,f6f400ac4a35dc9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:36.651 INFO 26087 --- [ocp-async-8,334d525c9aacdb23,d25750e3d2c4ef35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:36.727 INFO 26087 --- [ocp-schedules-11,186cf49c8b1ea743,59d4f4b55873b29a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:27:36.770 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:27:36.622674+08:00 2025-04-08 21:27:37.492 INFO 26087 --- [ocp-schedules-3,71513978f903a92e,ed5f0ce66297a9c1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:27:37.499 INFO 26087 --- [ocp-schedules-3,71513978f903a92e,ed5f0ce66297a9c1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:27:37.499 INFO 26087 --- [ocp-schedules-3,71513978f903a92e,ed5f0ce66297a9c1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:27:38.358 INFO 26087 --- [ocp-schedules-7,cbba99954935e2b5,b69e6a9c7291a16a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:27:38.359 INFO 26087 --- [ocp-schedules-7,cbba99954935e2b5,b69e6a9c7291a16a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:27:38.471 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:38.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:38.651 WARN 26087 --- [ocp-async-1,5b0a02629e7cd54c,a6068aa3c00fa700] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:38.651 INFO 26087 --- [ocp-async-4,b7e28405737ca87b,0cc624767777df84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:39.355 INFO 26087 --- [prometheus-minute-schedule-1,0a6b667f28883289,f89470ccdd24651b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:40.001 INFO 26087 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:27:40.001 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:27:40.651 WARN 26087 --- [ocp-async-6,86f32b4300de2ea1,5080cda6fe933079] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:40.651 INFO 26087 --- [ocp-async-3,a26fd0fb166b4bcf,37891c846f240b8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:40.767 INFO 26087 --- [ocp-schedules-21,3600cd246e60e7e2,a41b5431d4069c60] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:27:41.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:41.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:42.355 INFO 26087 --- [prometheus-minute-schedule-1,33088c02b3abe068,db8b3d7b915ae8c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:42.651 WARN 26087 --- [ocp-async-9,aab9aa1cd6fd7ae7,3d4df643ea3fcf57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:42.652 INFO 26087 --- [ocp-async-10,e557ff833511bce4,e803d414e9603882] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:44.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:44.473 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:44.652 WARN 26087 --- [ocp-async-1,0dc62b282b4fa713,2eb9b82b78ecc7c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:44.652 INFO 26087 --- [ocp-async-5,f2458605466a0c5b,6cfb6e232193a776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:45.355 INFO 26087 --- [prometheus-minute-schedule-1,ea04024558a14326,2471c051cca62146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:46.521 INFO 26087 --- [ocp-schedules-24,d35735f791d46636,9dcd224e67569c99] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:27:46.522 INFO 26087 --- [ocp-schedules-2,c1bb4b30849ba732,d38ef6dc402fb9e2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:27:46.522 INFO 26087 --- [ocp-schedules-2,c1bb4b30849ba732,d38ef6dc402fb9e2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:27:46.523 INFO 26087 --- [ocp-schedules-9,e782a98b551eeae6,3920c7666a92ecfd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:27:46.523 INFO 26087 --- [ocp-schedules-9,e782a98b551eeae6,3920c7666a92ecfd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:27:46.524 INFO 26087 --- [ocp-schedules-13,3e006cb1d283e879,eb1467c4b078f0aa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:27:46.524 INFO 26087 --- [ocp-schedules-13,3e006cb1d283e879,eb1467c4b078f0aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:27:46.524 INFO 26087 --- [ocp-schedules-15,db3ac58305764e8c,5332664fe0589e71] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:27:46.524 INFO 26087 --- [ocp-schedules-6,74336edc7c9db591,c7ac893083b7b23e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:27:46.524 INFO 26087 --- [ocp-schedules-10,89f6f35d1e08022f,46deb88b44b05d33] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:27:46.524 INFO 26087 --- [ocp-schedules-23,e6abbf5d439aef43,9207fdeee03e2a82] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:27:46.560 INFO 26087 --- [ocp-async-4,e9b42934e6762922,d80ec79e4b19353c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:27:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:27:46.623717+08:00, lastProcessTime=2025-04-08T21:27:36.622674+08:00 2025-04-08 21:27:46.630 INFO 26087 --- [ocp-schedules-20,24d108cc5bd1375b,4132d4a0982aef6f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:27:46.652 WARN 26087 --- [ocp-async-6,bd171f3de47bb474,7d39ca007a91fd8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:46.652 INFO 26087 --- [ocp-async-2,00bb224a8080dcd2,e460f5dc5c161ba8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:46.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:27:46.623717+08:00 2025-04-08 21:27:46.728 INFO 26087 --- [ocp-schedules-30,5dd1cd807c8255d7,0a2ffbaa8bd37e56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:27:47.473 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:47.474 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:48.355 INFO 26087 --- [prometheus-minute-schedule-1,5ac0b0f0acd8dfd5,4628c63242e2a5e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:48.364 INFO 26087 --- [ocp-schedules-30,e0bc2fcd611dfdca,e31bacd215fb1a50] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:27:48.366 INFO 26087 --- [ocp-schedules-30,e0bc2fcd611dfdca,e31bacd215fb1a50] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:27:48.652 WARN 26087 --- [ocp-async-2,065ea197e6caf279,9f6c6a1d9a6c3165] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:48.652 INFO 26087 --- [ocp-async-7,bd0d6e7f66adb7df,77f4a9f1a9b740eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:50.474 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:50.474 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:50.652 WARN 26087 --- [ocp-async-7,55e370073720dbd8,90803c5083c34a37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:50.652 INFO 26087 --- [ocp-async-8,379266a9904aeac8,ab3ba3613ab8b027] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:50.787 INFO 26087 --- [ocp-schedules-20,6f2e13a44a5c81ab,9dea8566abcb95fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:27:51.355 INFO 26087 --- [prometheus-minute-schedule-1,8068bd081ae86ec6,7724c788a40d07c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:52.652 WARN 26087 --- [ocp-async-6,45ecec03c59bc834,48405d31a9ab3e87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:52.652 INFO 26087 --- [ocp-async-3,4ecadf6275975593,04a48731e651a0fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:53.474 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:53.475 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:54.355 INFO 26087 --- [prometheus-minute-schedule-1,6ed228d73d8084e6,38f77d893eb0e037] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:54.652 WARN 26087 --- [ocp-async-2,0ee839322d3cdac0,6374f9b0fc7c0272] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:54.652 INFO 26087 --- [ocp-async-10,cf5acca20cfc4bee,c499ab8829ad8d43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:56.475 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:56.475 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:27:56.560 INFO 26087 --- [ocp-async-9,7f50e541e2ac5e87,0a8c9619ea36f7ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:27:56.576 INFO 26087 --- [ocp-schedules-10,0d285395e22275ec,b98e1be448ee5b69] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:27:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:27:56.622808+08:00, lastProcessTime=2025-04-08T21:27:46.623717+08:00 2025-04-08 21:27:56.652 INFO 26087 --- [ocp-async-7,6baeab19c1f316b4,af22258272fe1c53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:56.652 WARN 26087 --- [ocp-async-1,1a5a9927c73f6013,cc4ccb36c97fe3b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:56.730 INFO 26087 --- [ocp-schedules-25,a0a22b8a0e6edbde,6bc5266e904f95cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:27:56.759 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:27:56.622808+08:00 2025-04-08 21:27:57.026 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,,] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in method name [0x03...]. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:407) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:269) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:27:57.355 INFO 26087 --- [prometheus-minute-schedule-1,2d51f479ad4cc28f,84ba8727b3dc5b79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118880, interval=60 2025-04-08 21:27:57.500 INFO 26087 --- [ocp-schedules-27,9ed97698ec49eaa5,07ae8a2142aec914] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:27:57.520 INFO 26087 --- [ocp-schedules-27,9ed97698ec49eaa5,07ae8a2142aec914] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:27:57.520 INFO 26087 --- [ocp-schedules-27,9ed97698ec49eaa5,07ae8a2142aec914] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:27:58.371 INFO 26087 --- [ocp-schedules-29,0306d986298720c0,6e3d451a69bd0d25] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:27:58.372 INFO 26087 --- [ocp-schedules-29,0306d986298720c0,6e3d451a69bd0d25] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:27:58.653 WARN 26087 --- [ocp-async-5,1a27669a07793474,8d0eab24e3afb1aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:27:58.653 INFO 26087 --- [ocp-async-8,ea0018b874dab214,80a3bf0fe560391f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:27:59.476 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:27:59.476 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:00.355 INFO 26087 --- [prometheus-minute-schedule-1,180c728c4913d1c8,119559c619fddfb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:00.653 WARN 26087 --- [ocp-async-3,d2cadf5e12fbbbc0,59f70fbfd1ccd784] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:00.653 INFO 26087 --- [ocp-async-8,4d1c7a748880095d,0c7f45605f2c635f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:00.804 INFO 26087 --- [ocp-schedules-22,04ec7b59d3796a2c,a940db19a2536357] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:28:02.476 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:02.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:02.653 WARN 26087 --- [ocp-async-8,a79476650ffb970b,b5f96a983dc307c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:02.653 INFO 26087 --- [ocp-async-6,316ff0c951769e90,2f5c125d30f117e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:03.355 INFO 26087 --- [prometheus-minute-schedule-1,b7e8887029505e92,0d66c37a52eec886] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:04.653 WARN 26087 --- [ocp-async-7,d60b8ac1bb187ed7,42427886e59776dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:04.653 INFO 26087 --- [ocp-async-5,cbf2be3bdcfc2a71,cd047b4a2029fede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:05.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:05.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:06.355 INFO 26087 --- [prometheus-minute-schedule-1,51b2013bd4bde5c4,c8396d854780ce34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:06.355 INFO 26087 --- [prometheus-cache-monitor1,a99425fd67c2fc7c,8e3e90867e377eaa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:28:06.355 INFO 26087 --- [prometheus-cache-monitor1,a99425fd67c2fc7c,8e3e90867e377eaa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:28:06.560 INFO 26087 --- [ocp-async-6,32811bed457334e5,e8c99c3ec5d73c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:28:06.570 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:24:06.545, end=2025-04-08T21:26:06.569} 2025-04-08 21:28:06.574 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:28:06.574 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:28:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:28:06.622927+08:00, lastProcessTime=2025-04-08T21:27:56.622808+08:00 2025-04-08 21:28:06.653 INFO 26087 --- [ocp-async-10,cba37211c21932a6,f82568772411353b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:06.653 WARN 26087 --- [ocp-async-9,df844280885680d1,fe8fa0531529c5ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:06.726 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:28:06.622927+08:00 2025-04-08 21:28:06.731 INFO 26087 --- [ocp-schedules-13,06c7a235e209bea5,10138f9379d9485e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:28:08.378 INFO 26087 --- [ocp-schedules-28,7c10f4f16b4d1ac4,2ed8e0826be88346] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:28:08.379 INFO 26087 --- [ocp-schedules-28,7c10f4f16b4d1ac4,2ed8e0826be88346] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:28:08.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:08.478 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:08.653 WARN 26087 --- [ocp-async-1,e8bec06fb058ff76,894caea91be6c765] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:08.653 INFO 26087 --- [ocp-async-8,c202c08cabbd5b30,4f7fc998f5422d27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:09.355 INFO 26087 --- [prometheus-minute-schedule-1,dd600134688a213f,1b475c979284699e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:10.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:28:10.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:28:10.604 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:28:10.653 WARN 26087 --- [ocp-async-2,f4ff35f7e4fc23d1,de1f1bf0b1fbec42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:10.653 INFO 26087 --- [ocp-async-7,97c6d6e76f3df6a2,b557d33fdab04f01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:10.825 INFO 26087 --- [ocp-schedules-4,ceb3d06e8068cc3e,3dfa2bbbac3e8331] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:28:11.478 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:11.479 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:12.355 INFO 26087 --- [prometheus-minute-schedule-1,8b9e22b940143ab2,67bd426b7f6d12a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:12.481 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118820, nodeCount=378, affectRows=378 2025-04-08 21:28:12.483 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118820, nodeCount=377, validCount=377 2025-04-08 21:28:12.654 WARN 26087 --- [ocp-async-7,57429d1e383325a6,6361f87d58d63c46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:12.654 INFO 26087 --- [ocp-async-9,5097e81511fa6384,2b4d2354f3530bbc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:14.479 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:14.479 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:14.654 WARN 26087 --- [ocp-async-9,b2aaadd4e2eaf0ae,d508dca47e900169] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:14.654 INFO 26087 --- [ocp-async-1,01899bdb3745dfc2,d009e37dc1e92784] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:15.355 INFO 26087 --- [prometheus-minute-schedule-1,b75faf40375030f6,35173909cbd54172] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:16.517 INFO 26087 --- [ocp-schedules-20,86944edc94bc181e,860b80c6989e2478] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:28:16.517 INFO 26087 --- [ocp-schedules-10,a4647f3ea9b44a96,5b89e968907a6884] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:28:16.520 INFO 26087 --- [ocp-schedules-14,9f09420fd6fb3637,c213c52f115bb144] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:28:16.520 INFO 26087 --- [ocp-schedules-14,9f09420fd6fb3637,c213c52f115bb144] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:28:16.521 INFO 26087 --- [ocp-schedules-11,ba9369da0e3df9cc,ff1497121d968944] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:28:16.523 INFO 26087 --- [ocp-schedules-24,48ae09b306fe1c42,f7c19b276604f138] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:28:16.523 INFO 26087 --- [ocp-schedules-28,811d8f35b7935b9f,79ae28051ede9543] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:28:16.523 INFO 26087 --- [ocp-schedules-24,48ae09b306fe1c42,f7c19b276604f138] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:28:16.523 INFO 26087 --- [ocp-schedules-28,811d8f35b7935b9f,79ae28051ede9543] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:28:16.524 INFO 26087 --- [ocp-schedules-10,a4647f3ea9b44a96,5b89e968907a6884] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:28:16.524 INFO 26087 --- [ocp-schedules-3,846f39024752c34e,468295d6584e0b97] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:28:16.524 INFO 26087 --- [ocp-schedules-21,ee1d1da36f28910c,5f3ca7a7971d7d1d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:28:16.524 INFO 26087 --- [ocp-schedules-12,f3f169617b51f3e7,68dad1483de14980] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:28:16.532 INFO 26087 --- [ocp-schedules-20,86944edc94bc181e,860b80c6989e2478] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:28:16.536 INFO 26087 --- [ocp-schedules-13,96a3317d16276fe4,fb6f3d97560ddee7] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:28:16.536 INFO 26087 --- [ocp-schedules-13,96a3317d16276fe4,fb6f3d97560ddee7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:28:16.539 INFO 26087 --- [ocp-schedules-15,edf55451261512c9,5e406cdc1815fb8f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:28:16.539 INFO 26087 --- [ocp-schedules-1,dbdaf196e099e3a5,c34076c2ec51aad5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:28:16.554 INFO 26087 --- [pool-overview-container-1,f5e5294287fadae2,5099a51905890104] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:28:16.554 INFO 26087 --- [pool-overview-container-4,bc806a8c7337e99b,b83ec9012e8314d1] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:28:16.561 INFO 26087 --- [ocp-async-2,10c1c2b5dbf6802f,d68e64fa4ede56de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:28:16.562 INFO 26087 --- [ocp-schedules-8,f534a2fea10ceb7e,f74845b014856f68] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:28:16.563 INFO 26087 --- [ocp-schedules-8,f534a2fea10ceb7e,f74845b014856f68] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:28:16.604 INFO 26087 --- [ocp-schedules-29,6e523059371ebf24,e0be223893dc08c9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:28:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:28:16.623197+08:00, lastProcessTime=2025-04-08T21:28:06.622927+08:00 2025-04-08 21:28:16.631 INFO 26087 --- [ocp-schedules-19,e303e12e8a39038f,bfc190bc7fee0cb9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:28:16.641 INFO 26087 --- [ocp-schedules-11,117a35a1fdc1f4df,bd94148309bc049a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:28:16.641 INFO 26087 --- [ocp-schedules-11,117a35a1fdc1f4df,bd94148309bc049a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:28:16.653 INFO 26087 --- [ocp-schedules-24,b4f254c3e5d0f213,5215b74aafd54080] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:28:16.653+08:00, clusters=[] 2025-04-08 21:28:16.654 WARN 26087 --- [ocp-async-5,a630a77552678a0c,fcabe256ecebe6ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:16.654 INFO 26087 --- [ocp-async-7,81e450d0352d2709,0fec2bc2ee7d3f07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:16.654 INFO 26087 --- [ocp-schedules-3,7c3fd1bed6dcae58,1570f5efcaea6d0c] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:28:16.654 INFO 26087 --- [ocp-schedules-3,7c3fd1bed6dcae58,1570f5efcaea6d0c] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:28:16.655 INFO 26087 --- [ocp-schedules-28,fa784c864cf58240,59d60b18654a7637] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:28:16.658 INFO 26087 --- [ocp-schedules-13,b4b9d5c85405d84a,21c6832a2f315454] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:28:16.658+08:00, clusters=[] 2025-04-08 21:28:16.679 INFO 26087 --- [ocp-schedules-1,36249bb0d592cb91,3150c76d4ccb5be4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:28:16.679 INFO 26087 --- [ocp-schedules-1,36249bb0d592cb91,3150c76d4ccb5be4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:28:16.680 INFO 26087 --- [ocp-schedules-1,36249bb0d592cb91,3150c76d4ccb5be4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:28:16.680 INFO 26087 --- [ocp-schedules-1,36249bb0d592cb91,3150c76d4ccb5be4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:28:16.684 INFO 26087 --- [ocp-schedules-15,b36e535e9c111a1d,ddb9aff23844d20b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:28:16.685 INFO 26087 --- [ocp-schedules-15,b36e535e9c111a1d,ddb9aff23844d20b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:28:16.685 INFO 26087 --- [ocp-schedules-15,b36e535e9c111a1d,ddb9aff23844d20b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:28:16.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:28:16.623197+08:00 2025-04-08 21:28:16.732 INFO 26087 --- [ocp-schedules-22,b882ccfc5668f0d2,a063e57ccb041336] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:28:16.825 INFO 26087 --- [ocp-schedules-25,89869629a9a7877e,09c54370bad1d6f2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:28:16.825 INFO 26087 --- [ocp-schedules-25,89869629a9a7877e,09c54370bad1d6f2] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:28:16.827 INFO 26087 --- [ocp-schedules-25,89869629a9a7877e,09c54370bad1d6f2] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:28:16.834 INFO 26087 --- [ocp-schedules-17,5dfe5ea785bcd256,1f744f1b3ed3c626] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:28:16.841 INFO 26087 --- [ocp-schedules-18,855fc74d96452e23,8f612571b6899b7f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:28:16.848 INFO 26087 --- [ocp-schedules-8,c3100d9ad861b7b2,6f3558edd6e904b2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:28:16.869 INFO 26087 --- [ocp-schedules-7,3243667f55a0b6cc,0b31adbb159dece7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:28:16.869 INFO 26087 --- [ocp-schedules-7,3243667f55a0b6cc,0b31adbb159dece7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:28:16.872 INFO 26087 --- [ocp-schedules-7,3243667f55a0b6cc,0b31adbb159dece7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:28:16.872 INFO 26087 --- [ocp-schedules-7,3243667f55a0b6cc,0b31adbb159dece7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:28:16.874 INFO 26087 --- [ocp-schedules-7,3243667f55a0b6cc,0b31adbb159dece7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:28:16.874 INFO 26087 --- [ocp-schedules-7,3243667f55a0b6cc,0b31adbb159dece7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:28:17.159 INFO 26087 --- [ocp-schedules-16,142170ffd9331d42,a2ad805fce235a40] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:28:17.479 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:17.480 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:17.521 INFO 26087 --- [ocp-schedules-21,328702e4ef03e24a,6be3d87a37e83bce] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:28:17.531 INFO 26087 --- [ocp-schedules-21,328702e4ef03e24a,6be3d87a37e83bce] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:28:17.531 INFO 26087 --- [ocp-schedules-21,328702e4ef03e24a,6be3d87a37e83bce] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:28:17.577 INFO 26087 --- [ocp-schedules-12,961c49b85cad8007,709cb21b0fa472a1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:28:17.577 INFO 26087 --- [ocp-schedules-12,961c49b85cad8007,709cb21b0fa472a1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:28:18.355 INFO 26087 --- [prometheus-minute-schedule-1,bdf04faaa7fc57bf,e863031ba4548422] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:18.384 INFO 26087 --- [ocp-schedules-7,8457c25ce0df9ae2,1e70ab3a56b05ba0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:28:18.385 INFO 26087 --- [ocp-schedules-7,8457c25ce0df9ae2,1e70ab3a56b05ba0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:28:18.654 INFO 26087 --- [ocp-async-3,0241a1c577334f69,85c9a6ea2bf24e91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:18.654 WARN 26087 --- [ocp-async-8,3da9de8dd8bfdc23,83e56d1448696db4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:20.480 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:20.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:28:20.654 WARN 26087 --- [ocp-async-4,fd4da40ae07cb333,5f917094c5a838d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:20.654 INFO 26087 --- [ocp-async-2,bc6da14190134e42,2b23a05fa594c9f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:20.843 INFO 26087 --- [ocp-schedules-15,fe5fd4fababfdbb3,18792fef7559d3db] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:28:21.355 INFO 26087 --- [prometheus-minute-schedule-1,c94ae630759e2f92,cb83a4b171b5e2ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:22.654 WARN 26087 --- [ocp-async-6,75d8380da5cea7cc,ad5a845f3713332b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:22.654 INFO 26087 --- [ocp-async-5,b7feadbb39df932a,cde7c1b4cd54aba4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:23.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:23.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:24.355 INFO 26087 --- [prometheus-minute-schedule-1,f2f2724dd4f90b8b,179c4ce83775e5f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:24.655 WARN 26087 --- [ocp-async-1,eb7ed86037b79d15,4c4d84b5ab632cfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:24.655 INFO 26087 --- [ocp-async-8,c67c3eec19abcbde,22295a4fb3b588ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:26.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:26.482 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:26.561 INFO 26087 --- [ocp-async-9,fe3f501530e8e413,16cdf825bcf801be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:28:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:28:26.623264+08:00, lastProcessTime=2025-04-08T21:28:16.623197+08:00 2025-04-08 21:28:26.655 WARN 26087 --- [ocp-async-6,f4025597341814f5,d53fee211db4cf31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:26.655 INFO 26087 --- [ocp-async-3,c24aed2b847f6155,f587a65252784bf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:26.734 INFO 26087 --- [ocp-schedules-5,1f8d3f1fedf70628,84eda81c8d80a21a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:28:26.739 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:28:26.623264+08:00 2025-04-08 21:28:27.355 INFO 26087 --- [prometheus-minute-schedule-1,17d647603a40ad83,051fc35814610ef6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:28.392 INFO 26087 --- [ocp-schedules-25,fc6c5aebe7c540b3,00312a1fd4f826a6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:28:28.393 INFO 26087 --- [ocp-schedules-25,fc6c5aebe7c540b3,00312a1fd4f826a6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:28:28.655 WARN 26087 --- [ocp-async-1,878632621779d147,0a050445794daffa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:28.655 INFO 26087 --- [ocp-async-7,b66fa6db538de1e6,1bdde09fe35322c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:29.482 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:29.483 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:30.355 INFO 26087 --- [prometheus-minute-schedule-1,1c77dad74c308568,50fcf903bfe87f3b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:30.655 WARN 26087 --- [ocp-async-9,ca5048378a6ce687,1fbf3fe2a9889fdb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:30.655 INFO 26087 --- [ocp-async-5,898e096f5f8b763f,e9c54b152f3b14da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:30.881 INFO 26087 --- [ocp-schedules-3,117009a2dd3adfa5,8f759479b83be231] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:28:32.483 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:32.484 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:32.655 WARN 26087 --- [ocp-async-6,8de13d6f6b43cb4d,747e2d1b8457b4a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:32.655 INFO 26087 --- [ocp-async-5,f3c127bd3f549180,1d8fb0ccf696f22a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:33.355 INFO 26087 --- [prometheus-minute-schedule-1,d59c3eeb0129d670,e3e25fd74ccfe1ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:34.655 WARN 26087 --- [ocp-async-3,49b3a746362bf019,ffcc4a067f239031] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:34.655 INFO 26087 --- [ocp-async-10,74de4eb513922044,5be5602be044455c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:35.484 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:35.484 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:36.355 INFO 26087 --- [prometheus-minute-schedule-1,52ae85d7be1a723e,0a9669879cc40262] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:36.561 INFO 26087 --- [ocp-async-10,b26b51d4e8415358,7c3a7ae3144aa1ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:28:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:28:36.623374+08:00, lastProcessTime=2025-04-08T21:28:26.623264+08:00 2025-04-08 21:28:36.656 WARN 26087 --- [ocp-async-8,87da86423c16fca3,299210d8b5f6de1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:36.656 INFO 26087 --- [ocp-async-9,723352d1dbe05786,a624fe77422d240f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:28:36.623374+08:00 2025-04-08 21:28:36.735 INFO 26087 --- [ocp-schedules-14,dbb774d9bc238996,c224699bdd14492d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:28:37.533 INFO 26087 --- [ocp-schedules-9,a6109185ca3d9d40,0edf30c2f2cf7f3f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:28:37.537 INFO 26087 --- [ocp-schedules-9,a6109185ca3d9d40,0edf30c2f2cf7f3f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:28:37.537 INFO 26087 --- [ocp-schedules-9,a6109185ca3d9d40,0edf30c2f2cf7f3f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:28:38.088 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:03:07+08:00, updateTime=2025-04-08T21:28:07+08:00, expireTime=null, isLeader=false} 2025-04-08 21:28:38.090 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 21:28:38.404 INFO 26087 --- [ocp-schedules-15,e985f8a04c86b0ba,ea494370f047683c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:28:38.405 INFO 26087 --- [ocp-schedules-15,e985f8a04c86b0ba,ea494370f047683c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:28:38.484 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:38.485 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:38.656 WARN 26087 --- [ocp-async-2,383416ebc37a94e9,0862efef6eac3b85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:38.656 INFO 26087 --- [ocp-async-6,a8a652e63274d0bf,42076fd65c5a5180] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:39.355 INFO 26087 --- [prometheus-minute-schedule-1,4ab995d1c08a254c,8bef87fd469da20b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:40.002 INFO 26087 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:28:40.002 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:28:40.656 WARN 26087 --- [ocp-async-3,83d4b4e36770bc02,26b58c6c5926b0fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:40.656 INFO 26087 --- [ocp-async-1,146058f59ee47426,c1ae8f6900526226] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:40.900 INFO 26087 --- [ocp-schedules-23,7669081193313917,e507edb51a47abd6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:28:41.485 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:41.486 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:42.355 INFO 26087 --- [prometheus-minute-schedule-1,3e6f402e74470737,a445e7daeaeff9c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:42.656 WARN 26087 --- [ocp-async-7,4285d9d1786d4654,10d2b87495eaf498] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:42.656 INFO 26087 --- [ocp-async-4,993905490978ee9f,2c016ee47e6e4ada] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:44.486 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:44.486 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:44.656 WARN 26087 --- [ocp-async-2,44930d6eff096cb1,7f7d14b8ec01ee92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:44.656 INFO 26087 --- [ocp-async-4,9fd3008f1492858c,a834a5a9f2a12aa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:45.355 INFO 26087 --- [prometheus-minute-schedule-1,717659d8c8d96f89,a27545de1bbd103e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:46.521 INFO 26087 --- [ocp-schedules-10,3c87a7b8e90b8da5,87bb3bb8b756a603] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:28:46.521 INFO 26087 --- [ocp-schedules-3,cceb62ca615d1cb8,d6aa12b91418632a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:28:46.521 INFO 26087 --- [ocp-schedules-3,cceb62ca615d1cb8,d6aa12b91418632a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:28:46.523 INFO 26087 --- [ocp-schedules-24,108b1dd92a4b8d49,b203b59665b85e11] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:28:46.523 INFO 26087 --- [ocp-schedules-24,108b1dd92a4b8d49,b203b59665b85e11] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:28:46.523 INFO 26087 --- [ocp-schedules-6,3155ad7293c80116,69954795d72cfe26] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:28:46.523 INFO 26087 --- [ocp-schedules-6,3155ad7293c80116,69954795d72cfe26] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:28:46.524 INFO 26087 --- [ocp-schedules-12,9912bee492d0d9dc,ef9fc44860919298] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:28:46.524 INFO 26087 --- [ocp-schedules-8,26afce0f31dbb0f4,850cc3bc2b13b482] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:28:46.524 INFO 26087 --- [ocp-schedules-5,7b0a726cddd5fcd1,edd80d84ea458159] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:28:46.526 INFO 26087 --- [ocp-schedules-28,e07119a52e45d960,9130ccbb7b863839] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:28:46.561 INFO 26087 --- [ocp-async-6,d6f9d5ec20a42927,a3745b4a551cc003] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:28:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:28:46.623208+08:00, lastProcessTime=2025-04-08T21:28:36.623374+08:00 2025-04-08 21:28:46.633 INFO 26087 --- [ocp-schedules-1,185b1fdc58d37d4a,5d8fbc10631d71d4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:28:46.656 WARN 26087 --- [ocp-async-10,60ed2c88824b3a68,c55c35f02eeeade9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:46.656 INFO 26087 --- [ocp-async-3,7d78559d3bd2c12a,14ec1ddf9965df32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:46.717 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:28:46.623208+08:00 2025-04-08 21:28:46.737 INFO 26087 --- [ocp-schedules-9,024f89fc9b6c68b8,0ae20c1b68bdc81f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:28:47.487 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:47.487 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:48.355 INFO 26087 --- [prometheus-minute-schedule-1,b20f770669376f4a,0d34acabb6994b8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:48.409 INFO 26087 --- [ocp-schedules-18,025b672e951c63c1,741df880cb3cc1cb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:28:48.410 INFO 26087 --- [ocp-schedules-18,025b672e951c63c1,741df880cb3cc1cb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:28:48.656 WARN 26087 --- [ocp-async-7,cc81314b7f4a2fe2,354eda50ca645b1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:48.656 INFO 26087 --- [ocp-async-9,3b7a28c36473bacc,cd3ff08ae01a93e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:50.487 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:50.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:50.657 WARN 26087 --- [ocp-async-2,b548b203b91decb1,25602402b4270eb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:50.657 INFO 26087 --- [ocp-async-9,d427d86c098a8df7,f06fb166750f902b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:50.921 INFO 26087 --- [ocp-schedules-22,a1d3ab50ddee56c3,425976ed5c6f77a9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:28:51.355 INFO 26087 --- [prometheus-minute-schedule-1,eddfedbb11c86987,5a8a5aeb43830982] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:52.657 WARN 26087 --- [ocp-async-4,b9f15799fa7e8a26,48e2c1518468b955] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:52.657 INFO 26087 --- [ocp-async-9,8b7e6ab38c97ae99,1fee3bae51f91736] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:53.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:53.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:54.355 INFO 26087 --- [prometheus-minute-schedule-1,22e991d8a8293668,596afc89b95fbdde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:54.657 WARN 26087 --- [ocp-async-3,c6b341f5b86abeb6,f6b89d7f8f75a046] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:54.657 INFO 26087 --- [ocp-async-9,91d49736eb2da60d,1bcb592c0413e3b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:56.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:56.489 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:28:56.561 INFO 26087 --- [ocp-async-8,f103aa1f69453185,c19b73cb8e687eb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:28:56.577 INFO 26087 --- [ocp-schedules-27,d6a6e0404346b560,014c454f006a2300] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:28:56.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:28:56.624394+08:00, lastProcessTime=2025-04-08T21:28:46.623208+08:00 2025-04-08 21:28:56.657 WARN 26087 --- [ocp-async-2,2483225a0c7b6094,8fddc5f9edbc9655] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:56.657 INFO 26087 --- [ocp-async-6,ede4e81672f058af,133fb30236a7208e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:56.738 INFO 26087 --- [ocp-schedules-18,bad66bae8cda49ca,bbceba08999ac2c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:28:56.760 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:28:56.624394+08:00 2025-04-08 21:28:57.355 INFO 26087 --- [prometheus-minute-schedule-1,0884175c9516c113,1864ad25997b33c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744118940, interval=60 2025-04-08 21:28:57.538 INFO 26087 --- [ocp-schedules-12,2e11d4aed5611e4f,4737dbe0d7cd8a22] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:28:57.553 INFO 26087 --- [ocp-schedules-12,2e11d4aed5611e4f,4737dbe0d7cd8a22] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:28:57.553 INFO 26087 --- [ocp-schedules-12,2e11d4aed5611e4f,4737dbe0d7cd8a22] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:28:58.413 INFO 26087 --- [ocp-schedules-30,c377d2adca8e1bba,e8a3dc961a9a169c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:28:58.414 INFO 26087 --- [ocp-schedules-30,c377d2adca8e1bba,e8a3dc961a9a169c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:28:58.657 WARN 26087 --- [ocp-async-10,c37b399f50419461,1af1224db10f44f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:28:58.657 INFO 26087 --- [ocp-async-1,6a24177df8ed4347,7448700a2c7ef1b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:28:59.489 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:28:59.490 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:00.355 INFO 26087 --- [prometheus-minute-schedule-1,4d179bcb4d6f4fb2,21240ec6797bf391] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:00.657 WARN 26087 --- [ocp-async-9,80b52e28bd047086,a7b7c49a99e7fb4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:00.657 INFO 26087 --- [ocp-async-7,c4508c913eda20bf,7431adb5beb6fc15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:00.937 INFO 26087 --- [ocp-schedules-30,1da279e6f4422dce,64c3cc041eab5a18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:29:02.490 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:02.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:02.657 WARN 26087 --- [ocp-async-5,5cc1a05faba62292,b2a0a9aca58f0171] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:02.657 INFO 26087 --- [ocp-async-4,600a47911c2b15cd,1a18ef96bd7718a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:03.355 INFO 26087 --- [prometheus-minute-schedule-1,2327c7b681623c16,13f9ecb636037a27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:04.658 WARN 26087 --- [ocp-async-6,027ac33ea3d0c2c8,91bc7f57b1d1e8e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:04.658 INFO 26087 --- [ocp-async-4,1a3b52174a79066f,d55b1db6aa68e07a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:05.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:05.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:06.330 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:23:06.314, end=2025-04-08T21:28:06.328} 2025-04-08 21:29:06.332 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:29:06.332 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:29:06.344 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:23:06.321, end=2025-04-08T21:28:06.332} 2025-04-08 21:29:06.346 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:29:06.346 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:29:06.355 INFO 26087 --- [prometheus-minute-schedule-1,a25ea81041130498,6fa227a49efdf01e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:06.355 INFO 26087 --- [prometheus-cache-monitor1,327de89f2856199c,ec18f516efe63a26] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:29:06.355 INFO 26087 --- [prometheus-cache-monitor1,327de89f2856199c,ec18f516efe63a26] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:29:06.562 INFO 26087 --- [ocp-async-10,d133e19f5424f860,600c9297ce6b3256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:29:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:29:06.623940+08:00, lastProcessTime=2025-04-08T21:28:56.624394+08:00 2025-04-08 21:29:06.658 INFO 26087 --- [ocp-async-8,5cde0089cfd03d25,8f5e1d8e68a186e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:06.658 WARN 26087 --- [ocp-async-1,15f869d8cc52a6dd,7df5d399194df206] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:06.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:29:06.623940+08:00 2025-04-08 21:29:06.740 INFO 26087 --- [ocp-schedules-1,f25d389f9f0f5937,a95889de6ba894cc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:29:08.417 INFO 26087 --- [ocp-schedules-20,4d58f85ce55c828a,a1716d14052ef922] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:29:08.418 INFO 26087 --- [ocp-schedules-20,4d58f85ce55c828a,a1716d14052ef922] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:29:08.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:08.492 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:08.658 WARN 26087 --- [ocp-async-7,3670ca5080d542ca,c7f9e89f86ca29fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:08.658 INFO 26087 --- [ocp-async-5,fb182aff783f3cad,af8db8d18422172b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:09.355 INFO 26087 --- [prometheus-minute-schedule-1,3304d477a78b1cfc,1a2639c9a0206f40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:10.001 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:29:10.001 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:29:10.602 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:29:10.658 INFO 26087 --- [ocp-async-5,4f6f0a21d8702e2e,66f067f870d5e705] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:10.658 WARN 26087 --- [ocp-async-3,9879d9c7242fcb72,cef7f37980ef8322] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:10.961 INFO 26087 --- [ocp-schedules-4,49eb48de83a47ed5,ac7fe2f42008bc79] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:29:11.492 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:11.493 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:12.355 INFO 26087 --- [prometheus-minute-schedule-1,6883c50a1fc28236,455f1077d25e20ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:12.485 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118880, nodeCount=378, affectRows=378 2025-04-08 21:29:12.487 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118880, nodeCount=377, validCount=377 2025-04-08 21:29:12.658 WARN 26087 --- [ocp-async-1,b8b7bf3d5e20ed84,002b39cdd3355331] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:12.658 INFO 26087 --- [ocp-async-4,c41c855e220e8bb1,73d009de1960517d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:14.493 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:14.493 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:14.658 WARN 26087 --- [ocp-async-4,39b11c18e30ad5dd,d3958f4eda51b334] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:14.658 INFO 26087 --- [ocp-async-7,5600cf7408fb3fab,7de9405173da1f29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:15.355 INFO 26087 --- [prometheus-minute-schedule-1,497dd3c9748d679b,96c3d8be23158471] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:16.517 INFO 26087 --- [ocp-schedules-3,090de8d5eda8e879,b538e1ed90ddfca6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:29:16.517 INFO 26087 --- [ocp-schedules-22,b4aaee238a33b860,05ba2cda48633d0d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:29:16.521 INFO 26087 --- [ocp-schedules-27,28ab73eaeb29ef73,771727f1780ffe04] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:29:16.521 INFO 26087 --- [ocp-schedules-27,28ab73eaeb29ef73,771727f1780ffe04] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:29:16.521 INFO 26087 --- [ocp-schedules-16,4b145e103915a911,275d60198346ccfa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:29:16.523 INFO 26087 --- [ocp-schedules-18,b7162c9501486783,41e5499116421ae3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:29:16.523 INFO 26087 --- [ocp-schedules-18,b7162c9501486783,41e5499116421ae3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:29:16.523 INFO 26087 --- [ocp-schedules-1,245758331ab6ae25,12673e7a63b96ce0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:29:16.524 INFO 26087 --- [ocp-schedules-1,245758331ab6ae25,12673e7a63b96ce0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:29:16.524 INFO 26087 --- [ocp-schedules-22,b4aaee238a33b860,05ba2cda48633d0d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:29:16.524 INFO 26087 --- [ocp-schedules-12,0a680eb4a23c4371,2e37c4eb8267a3b7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:29:16.524 INFO 26087 --- [ocp-schedules-28,53d6ac2ffb26be7c,0137ab51ec1d41c7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:29:16.524 INFO 26087 --- [ocp-schedules-8,dcc888b8c4a0f616,3eecc0f7fca36172] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:29:16.525 INFO 26087 --- [ocp-schedules-3,090de8d5eda8e879,b538e1ed90ddfca6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:29:16.528 INFO 26087 --- [ocp-schedules-27,d47add030417750e,359e6199a549b74d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:29:16.562 INFO 26087 --- [ocp-async-7,d05713f7a9b370d7,0f4973325523cca2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:29:16.586 INFO 26087 --- [ocp-schedules-2,d144379f90e1445d,f2b764a15fc54ea3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:29:16.587 INFO 26087 --- [ocp-schedules-2,d144379f90e1445d,f2b764a15fc54ea3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:29:16.605 INFO 26087 --- [ocp-schedules-24,a00a696d20bdc176,5c78a4b15dabb2e3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:29:16.619 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:29:16.619 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:29:16.624 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:29:16.624 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:29:16.624 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:29:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:29:16.623815+08:00, lastProcessTime=2025-04-08T21:29:06.623940+08:00 2025-04-08 21:29:16.625 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:29:16.625 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:29:16.625 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:29:16.628 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:29:16.628 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:29:16.628 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:29:16.630 INFO 26087 --- [ocp-schedules-25,239100484f948ddf,3a5fc418fc91303a] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:29:16.634 INFO 26087 --- [ocp-schedules-13,20960be184d10578,d4c0849d44e930d9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:29:16.643 INFO 26087 --- [ocp-schedules-30,b36d3aadbfab246e,06bfd9171ff48e55] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:29:16.643 INFO 26087 --- [ocp-schedules-30,b36d3aadbfab246e,06bfd9171ff48e55] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:29:16.654 INFO 26087 --- [ocp-schedules-23,d47fbab9a3c85c26,5a677746ba3c874e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:29:16.654+08:00, clusters=[] 2025-04-08 21:29:16.656 INFO 26087 --- [ocp-schedules-4,a7f82482aac31265,05fbbd1fe0f778b0] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:29:16.656 INFO 26087 --- [ocp-schedules-4,a7f82482aac31265,05fbbd1fe0f778b0] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:29:16.656 INFO 26087 --- [ocp-schedules-26,8af6ff39a14791c5,cd0003a17c21bf4f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:29:16.658 WARN 26087 --- [ocp-async-10,bfe72306693cfe2f,10b75e99d86e2511] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:16.659 INFO 26087 --- [ocp-async-1,aeb70f61c9b02b7f,0453711784a48ca6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:16.659 INFO 26087 --- [ocp-schedules-16,307c394e3f085e4e,e6896a17ea806440] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:29:16.659+08:00, clusters=[] 2025-04-08 21:29:16.681 INFO 26087 --- [ocp-schedules-12,79285e42d47640fe,899892428c477b48] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:29:16.681 INFO 26087 --- [ocp-schedules-12,79285e42d47640fe,899892428c477b48] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:29:16.682 INFO 26087 --- [ocp-schedules-12,79285e42d47640fe,899892428c477b48] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:29:16.682 INFO 26087 --- [ocp-schedules-12,79285e42d47640fe,899892428c477b48] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:29:16.685 INFO 26087 --- [ocp-schedules-28,36ef8ae7bf1219df,9dbc1fa79a615615] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:29:16.686 INFO 26087 --- [ocp-schedules-28,36ef8ae7bf1219df,9dbc1fa79a615615] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:29:16.686 INFO 26087 --- [ocp-schedules-28,36ef8ae7bf1219df,9dbc1fa79a615615] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:29:16.721 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:29:16.623815+08:00 2025-04-08 21:29:16.741 INFO 26087 --- [ocp-schedules-29,11d8c2f88e296ea6,7c90ea4bd47f441d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:29:16.834 INFO 26087 --- [ocp-schedules-9,7bdd9c2e423f692e,b5bbb3c7c8647587] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:29:16.835 INFO 26087 --- [ocp-schedules-20,4fb415a016e2425f,82b2ddaec039bc70] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:29:16.847 INFO 26087 --- [ocp-schedules-11,0af273b63974c85d,1e42f16edd5ed0c3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:29:16.849 INFO 26087 --- [ocp-schedules-15,88c9696967a61384,041ef234be3db910] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:29:16.877 INFO 26087 --- [ocp-schedules-6,7bc0dd7f9d4d236e,478a960fc638e7e5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:29:16.877 INFO 26087 --- [ocp-schedules-6,7bc0dd7f9d4d236e,478a960fc638e7e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:29:16.877 INFO 26087 --- [ocp-schedules-6,7bc0dd7f9d4d236e,478a960fc638e7e5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:29:16.877 INFO 26087 --- [ocp-schedules-6,7bc0dd7f9d4d236e,478a960fc638e7e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:29:16.877 INFO 26087 --- [ocp-schedules-6,7bc0dd7f9d4d236e,478a960fc638e7e5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:29:16.877 INFO 26087 --- [ocp-schedules-6,7bc0dd7f9d4d236e,478a960fc638e7e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:29:17.494 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:17.494 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:17.554 INFO 26087 --- [ocp-schedules-4,08934c01741d6b85,cf413abaddfaddbf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:29:17.563 INFO 26087 --- [ocp-schedules-4,08934c01741d6b85,cf413abaddfaddbf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:29:17.563 INFO 26087 --- [ocp-schedules-4,08934c01741d6b85,cf413abaddfaddbf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:29:17.580 INFO 26087 --- [ocp-schedules-18,b1a405588a3c1862,22c5ef813c581a25] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:29:17.580 INFO 26087 --- [ocp-schedules-18,b1a405588a3c1862,22c5ef813c581a25] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:29:18.355 INFO 26087 --- [prometheus-minute-schedule-1,769e70d24ab7d6cf,58e5b53dc3fc852b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:18.424 INFO 26087 --- [ocp-schedules-17,6d46b4b8f2a61e02,4f53094f30af2961] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:29:18.425 INFO 26087 --- [ocp-schedules-17,6d46b4b8f2a61e02,4f53094f30af2961] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:29:18.659 WARN 26087 --- [ocp-async-8,9b0c0a0208e11d49,3f606904a2ee74c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:18.659 INFO 26087 --- [ocp-async-6,85e0d12d596c61d4,b363be0f707a96d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:20.494 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:20.495 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:20.591 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:29:20.659 WARN 26087 --- [ocp-async-6,7034835f5c69dc83,16563f228d318d46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:20.659 INFO 26087 --- [ocp-async-7,6c5fdc08fdbd8b5d,ad8e083d92e8101c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:20.980 INFO 26087 --- [ocp-schedules-16,056f4b13ec31ff3f,a2b0094dbce6b04b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:29:21.355 INFO 26087 --- [prometheus-minute-schedule-1,37cc7d43a227fd84,ea0a4829868d6dac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:22.659 INFO 26087 --- [ocp-async-9,5bbe92562d6dc55e,d3dc6aacbd50d9d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:22.659 WARN 26087 --- [ocp-async-10,04b787c3bdd94d8c,8758fe15531aa5d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:23.495 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:23.496 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:24.355 INFO 26087 --- [prometheus-minute-schedule-1,02f878963035eba4,e8bc7e489f03ff23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:24.659 WARN 26087 --- [ocp-async-8,80651ff7332601da,46358b70ea22c890] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:24.659 INFO 26087 --- [ocp-async-4,28246516f36172a2,67d4312ff69d00a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:26.496 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:26.497 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:26.562 INFO 26087 --- [ocp-async-3,7af4e03dac031570,bf3da3ee94f16ca4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:29:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:29:26.624500+08:00, lastProcessTime=2025-04-08T21:29:16.623815+08:00 2025-04-08 21:29:26.659 WARN 26087 --- [ocp-async-2,431565cdfde96819,1d378e55e9a796d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:26.659 INFO 26087 --- [ocp-async-10,e626f070c76e22ef,a061903b054330bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:26.742 INFO 26087 --- [ocp-schedules-20,e29fc8de1cbb3000,fc04689a1922f47b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:29:26.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:29:26.624500+08:00 2025-04-08 21:29:27.355 INFO 26087 --- [prometheus-minute-schedule-1,801704323e79320e,009ef0dcf86a657a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:28.430 INFO 26087 --- [ocp-schedules-19,5f894899221faead,ccd4926f8d97ba89] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:29:28.432 INFO 26087 --- [ocp-schedules-19,5f894899221faead,ccd4926f8d97ba89] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:29:28.659 WARN 26087 --- [ocp-async-10,31d3a80fe292e488,3ba6b0b46ebefead] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:28.659 INFO 26087 --- [ocp-async-8,cb191557cca04728,083939d583c7f68d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:29.497 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:29.498 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:30.355 INFO 26087 --- [prometheus-minute-schedule-1,31eebad66e3684a2,df1c1c998cdb3221] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:30.659 WARN 26087 --- [ocp-async-4,3f922baa05c2a24c,5cefbccafcb0a1e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:30.659 INFO 26087 --- [ocp-async-3,d008cf7cd163c345,704e11f9e308500d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:31.001 INFO 26087 --- [ocp-schedules-19,94696de7bf3ceb24,9cddaf9089c790a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:29:32.498 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:32.498 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:32.660 WARN 26087 --- [ocp-async-3,edfcb1f885b37f26,8995bcc1cda07d2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:32.660 INFO 26087 --- [ocp-async-7,9b933057c422787d,0c4b9c9c34444bf9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:33.355 INFO 26087 --- [prometheus-minute-schedule-1,b9d3fe63948aa724,cf7c124d3f91bda6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:34.660 WARN 26087 --- [ocp-async-10,cf3fa32e7629d5c1,8c3e8175d94ce18e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:34.660 INFO 26087 --- [ocp-async-5,6bd426997c4776d9,77c9a4d82a6d1ea1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:34.978 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:29:34.979 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:29:35.499 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:35.499 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:36.355 INFO 26087 --- [prometheus-minute-schedule-1,a2fb0ed076ec60de,4e0148f605e3df35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:36.562 INFO 26087 --- [ocp-async-8,5e2a536c04c6eae7,81d605e8e0b6fb56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:29:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:29:36.624065+08:00, lastProcessTime=2025-04-08T21:29:26.624500+08:00 2025-04-08 21:29:36.660 INFO 26087 --- [ocp-async-4,2c77c94e07b520c1,b5c8d8682bd5dc2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:36.660 WARN 26087 --- [ocp-async-6,5de62311448b3789,482aa285301154e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:36.725 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:29:36.624065+08:00 2025-04-08 21:29:36.744 INFO 26087 --- [ocp-schedules-11,6131fa79a9ec3591,dc5448d5a7efdbe4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:29:37.564 INFO 26087 --- [ocp-schedules-26,bb10aa923f6d4bf8,d8d0dd7b9747126e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:29:37.567 INFO 26087 --- [ocp-schedules-26,bb10aa923f6d4bf8,d8d0dd7b9747126e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:29:37.567 INFO 26087 --- [ocp-schedules-26,bb10aa923f6d4bf8,d8d0dd7b9747126e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:29:38.438 INFO 26087 --- [ocp-schedules-6,2063284668604b0d,6fe2fa974350363e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:29:38.438 INFO 26087 --- [ocp-schedules-6,2063284668604b0d,6fe2fa974350363e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:29:38.499 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:38.500 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:38.660 WARN 26087 --- [ocp-async-2,32686a6c82f75c94,7fa51c9b4590c039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:38.660 INFO 26087 --- [ocp-async-3,41bfc2d28d122f9e,6241d09c888bb582] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:39.355 INFO 26087 --- [prometheus-minute-schedule-1,df495f659a668f81,25ae83b20ffa6922] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:40.002 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:29:40.002 INFO 26087 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:29:40.660 WARN 26087 --- [ocp-async-10,8eb27ddfd6ce7ad8,eadfc3809a500f77] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:40.660 INFO 26087 --- [ocp-async-9,20dc2715585c4fff,2ede9a2cd9dcd9cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:41.017 INFO 26087 --- [ocp-schedules-30,d7caa636864c02c1,3acf6ff6e0672d18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:29:41.500 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:41.501 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:42.355 INFO 26087 --- [prometheus-minute-schedule-1,bed7d93d8a1909f2,5dbd71a32a4a3c23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:42.660 WARN 26087 --- [ocp-async-6,6d752d1eb25ce50d,b9ae2d2de4c47a48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:42.660 INFO 26087 --- [ocp-async-1,ff65d6597749f0a9,85d6081d793431de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:44.501 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:44.502 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:44.661 WARN 26087 --- [ocp-async-4,fe078b8662f5d09a,db814296890c41df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:44.661 INFO 26087 --- [ocp-async-7,ca14ae0b53e5c147,6c988033b70124d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:45.355 INFO 26087 --- [prometheus-minute-schedule-1,7a45b664a0136d7c,76c513020bd5031e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:46.521 INFO 26087 --- [ocp-schedules-20,dc952ea999623396,027a8ff6a7b4cb8a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:29:46.522 INFO 26087 --- [ocp-schedules-23,4ff3f219bcf2897d,64bee6188c71310d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:29:46.522 INFO 26087 --- [ocp-schedules-23,4ff3f219bcf2897d,64bee6188c71310d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:29:46.523 INFO 26087 --- [ocp-schedules-22,8bb8123e2cce6798,5c5ce8dbe351537e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:29:46.523 INFO 26087 --- [ocp-schedules-22,8bb8123e2cce6798,5c5ce8dbe351537e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:29:46.523 INFO 26087 --- [ocp-schedules-12,b9573e6bb7a4400d,dbd435feaf75b7a7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:29:46.523 INFO 26087 --- [ocp-schedules-12,b9573e6bb7a4400d,dbd435feaf75b7a7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:29:46.524 INFO 26087 --- [ocp-schedules-1,42b1722d697e2993,378022fe741bdba4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:29:46.524 INFO 26087 --- [ocp-schedules-16,bd4fd6184a8dc6ec,1e62fdb8f8e870fc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:29:46.524 INFO 26087 --- [ocp-schedules-13,1f79e8be55863555,f3d1b886b7626862] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:29:46.535 INFO 26087 --- [ocp-schedules-28,02a991067ddb3570,636148b679b84881] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:29:46.562 INFO 26087 --- [ocp-async-3,383d1443890a98e7,f5856f800c11e5af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:29:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:29:46.623168+08:00, lastProcessTime=2025-04-08T21:29:36.624065+08:00 2025-04-08 21:29:46.635 INFO 26087 --- [ocp-schedules-19,850e5d30497f6edd,388f61f194bd6c5f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:29:46.661 WARN 26087 --- [ocp-async-10,8f4007d1f24ee6cf,bd879f0a8c86d1c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:46.661 INFO 26087 --- [ocp-async-9,c508b893383c00b1,b0fb82927c105fe7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:46.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:29:46.623168+08:00 2025-04-08 21:29:46.746 INFO 26087 --- [ocp-schedules-9,403d8d0588f2a754,0669d8773facd143] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:29:47.502 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:47.502 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:48.355 INFO 26087 --- [prometheus-minute-schedule-1,0d370575623cb747,20cd75764f76f3f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:48.445 INFO 26087 --- [ocp-schedules-7,f4676035b73a63b6,662ebbbc85aaac3c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:29:48.446 INFO 26087 --- [ocp-schedules-7,f4676035b73a63b6,662ebbbc85aaac3c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:29:48.661 WARN 26087 --- [ocp-async-6,857cbb42257ba1f3,1f8265e672f6acd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:48.661 INFO 26087 --- [ocp-async-5,cdf34190fd728f47,fac630cbb61d3a60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:50.502 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:50.503 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:50.661 WARN 26087 --- [ocp-async-4,7e2e617b12b1ea9d,ed49b6c4bb1e7b91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:50.661 INFO 26087 --- [ocp-async-2,4d2a8b7e147dad80,d2e14d133f8edfc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:51.040 INFO 26087 --- [ocp-schedules-1,4edec5f811302fc1,e6757e91d28807dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:29:51.355 INFO 26087 --- [prometheus-minute-schedule-1,70815e269baed22c,142cfeb2c3d74e10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:52.661 WARN 26087 --- [ocp-async-2,37afd953751977a5,d7459dfe33352762] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:52.661 INFO 26087 --- [ocp-async-7,beb729c8daec5c8e,60a0e53202d006c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:53.503 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:53.504 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:54.355 INFO 26087 --- [prometheus-minute-schedule-1,9df2e1201f92e103,a09b83006fa36665] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:54.661 WARN 26087 --- [ocp-async-6,ce6fa1a02ace31c4,32befebf720fd092] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:54.661 INFO 26087 --- [ocp-async-7,b241d984635c3daa,75c12d976f47044d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:56.504 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:56.504 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:29:56.562 INFO 26087 --- [ocp-async-9,ee710117e1b7987e,7f89512096ad483f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:29:56.578 INFO 26087 --- [ocp-schedules-16,1a38aea617745ad9,7b300f7162aaa49c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:29:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:29:56.623279+08:00, lastProcessTime=2025-04-08T21:29:46.623168+08:00 2025-04-08 21:29:56.662 WARN 26087 --- [ocp-async-3,f8bf00d6d18d509f,e963728138d3cf48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:56.661 INFO 26087 --- [ocp-async-5,ba33642a335fddcb,c61eadb12a351016] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:56.721 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:29:56.623279+08:00 2025-04-08 21:29:56.747 INFO 26087 --- [ocp-schedules-5,07787c9aa9ea50e0,7bbae5e0c851861d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:29:57.355 INFO 26087 --- [prometheus-minute-schedule-1,34dc96876d99c64c,1725aaf0bffdcfb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119000, interval=60 2025-04-08 21:29:57.569 INFO 26087 --- [ocp-schedules-9,5c016c7e53d12634,18c0553d0a6b2b35] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:29:57.577 INFO 26087 --- [ocp-schedules-9,5c016c7e53d12634,18c0553d0a6b2b35] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:29:57.577 INFO 26087 --- [ocp-schedules-9,5c016c7e53d12634,18c0553d0a6b2b35] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:29:58.450 INFO 26087 --- [ocp-schedules-30,0dd6399ed68cb7f7,29af83d2fa06d576] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:29:58.451 INFO 26087 --- [ocp-schedules-30,0dd6399ed68cb7f7,29af83d2fa06d576] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:29:58.662 INFO 26087 --- [ocp-async-5,5754121a3bc56b85,c2ee26b81c3b6abe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:29:58.662 WARN 26087 --- [ocp-async-10,33b327937d4b6eb1,d782ae96679bf864] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:29:59.505 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:29:59.505 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:00.355 INFO 26087 --- [prometheus-minute-schedule-1,098c2995dd3d2700,0ca3333d9f39fbc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:00.553 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:25:01.0, nextRun=2025-04-08 21:30:00.0, currentDate=Tue Apr 08 21:30:00 CST 2025 2025-04-08 21:30:00.554 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:25:01.0, nextRunTime=2025-04-08 21:30:00.0} 2025-04-08 21:30:00.556 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:30:00.556 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:30:00.556+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:30:00.564 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:25:01.0, nextRun=2025-04-08 21:30:00.0, currentDate=Tue Apr 08 21:30:00 CST 2025 2025-04-08 21:30:00.564 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:25:01.0, nextRunTime=2025-04-08 21:30:00.0} 2025-04-08 21:30:00.565 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:30:00.567 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:30:00.567 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,5c39feae89dfaf3f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:30:00.662 WARN 26087 --- [ocp-async-5,71269e0b6720558e,be341f96267eee98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:00.662 INFO 26087 --- [ocp-async-1,c401d1da21e217ab,f77632959179128f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:01.058 INFO 26087 --- [ocp-schedules-20,05c62b7f6eb6ce3b,ae54d0fd233e36dd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:30:01.291 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,ac66fae3098b0ae5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000123, operation:EXECUTE, state: RUNNING 2025-04-08 21:30:01.323 INFO 26087 --- [subtask-executor11,8ec1a44aeefd3a62,210fecc76dc7adb4] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:30:01.323 INFO 26087 --- [subtask-executor11,8ec1a44aeefd3a62,210fecc76dc7adb4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000123, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000014, task_operation=execute, latest_execution_start_time=2025-04-08T21:30:01.287+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000123}, listMap={}}, executor=192.168.1.61 2025-04-08 21:30:01.329 INFO 26087 --- [subtask-executor11,8ec1a44aeefd3a62,210fecc76dc7adb4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000123, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:30:01.627 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000014, status:SUCCESS 2025-04-08 21:30:01.627 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000014, status:SUCCESS 2025-04-08 21:30:02.505 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:02.506 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:02.662 WARN 26087 --- [ocp-async-1,be15c684abd02c5b,bbd0526f48cb9afc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:02.662 INFO 26087 --- [ocp-async-3,8d14c0a31055692e,4eff7956f72f9bf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:03.355 INFO 26087 --- [prometheus-minute-schedule-1,d1713c2fecfe53a9,ddfdc3dade79b4ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:04.639 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,04036a61bf651caf] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:20:04.0, nextRun=2025-04-08 21:30:04.0, currentDate=Tue Apr 08 21:30:04 CST 2025 2025-04-08 21:30:04.639 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,04036a61bf651caf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-04-08 21:20:04.0, nextRunTime=2025-04-08 21:30:04.0} 2025-04-08 21:30:04.642 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,04036a61bf651caf] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 21:30:04.642 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,04036a61bf651caf] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:30:04.642+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:30:04.662 WARN 26087 --- [ocp-async-3,3868dc63839e1c01,80786f8ddf591b09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:04.662 INFO 26087 --- [ocp-async-10,13ebdc5dab19461e,66500b9cc2a75a19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:05.312 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,17540096959f1765] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000124, operation:EXECUTE, state: RUNNING 2025-04-08 21:30:05.336 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:30:05.337 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000124, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000015, task_operation=execute, latest_execution_start_time=2025-04-08T21:30:05.310+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=1000124}, listMap={}}, executor=192.168.1.61 2025-04-08 21:30:05.354 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:30:05.354 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:30:05.355 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.355 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:30:05.356 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.356 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.356 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.357 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:30:05.368 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.369 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.369 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.369 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:30:05.379 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.380 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.380 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.380 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:30:05.390 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.391 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.391 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.391 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:30:05.402 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.402 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.402 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.402 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:30:05.406 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.407 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.407 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.407 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:30:05.410 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.411 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.411 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.411 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:30:05.414 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.415 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.415 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.415 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:30:05.419 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.420 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.420 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.420 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:30:05.424 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.424 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.425 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.425 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:30:05.429 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.430 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.430 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.430 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:30:05.437 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.438 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.438 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.438 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:30:05.449 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.449 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.450 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.450 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:30:05.454 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.454 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.454 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.454 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:30:05.459 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.459 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.459 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.460 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:30:05.463 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.464 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.464 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.464 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:30:05.467 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.467 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.468 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.468 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:30:05.471 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.471 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.472 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.472 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:30:05.475 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.475 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.476 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.476 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:30:05.479 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.479 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.480 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.480 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:30:05.483 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.483 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.484 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.484 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:30:05.487 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.488 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.488 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.488 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:30:05.491 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.491 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.492 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.492 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:30:05.495 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.495 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.496 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.496 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:30:05.499 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.499 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.500 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.500 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:30:05.503 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.503 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.503 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.504 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:30:05.506 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:05.506 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:05.507 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.507 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.508 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.508 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:30:05.511 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.511 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.511 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.512 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:30:05.515 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.515 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.515 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.516 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:30:05.519 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.519 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.519 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.520 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:30:05.523 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.523 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.523 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.523 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:30:05.526 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.527 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.527 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.527 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:30:05.530 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.531 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.531 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.531 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:30:05.534 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.534 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.534 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.534 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:30:05.540 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.540 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.541 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.541 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:30:05.546 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.546 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.546 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.547 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:30:05.551 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.551 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.551 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.551 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:30:05.556 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.556 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.556 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.557 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:30:05.560 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.560 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.560 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.560 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:30:05.563 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.564 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.564 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.564 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:30:05.567 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.567 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.567 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.567 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:30:05.571 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.571 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.572 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.572 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:30:05.579 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.579 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.580 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.580 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:30:05.596 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.597 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.597 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.598 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:30:05.603 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.603 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.603 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.604 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:30:05.609 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.609 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.610 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.610 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:30:05.615 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.615 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.615 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.615 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:30:05.618 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.619 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.619 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.619 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:30:05.622 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.622 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.623 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.623 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:30:05.626 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:30:05.626 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:30:05.627 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.627 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:30:05.628 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.628 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.628 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.629 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:30:05.635 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.635 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:30:05.635 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:30:05.635 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:30:05.639 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:30:05.658 INFO 26087 --- [subtask-executor12,8ec1a44aeefd3a62,6501ed16088d6bd9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000124, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:30:06.355 INFO 26087 --- [prometheus-minute-schedule-1,8d8dfd295f0773b2,a3c3c45c93dd02af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:06.355 INFO 26087 --- [prometheus-cache-monitor1,63782ef66b0202de,5142754809d54b9c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:30:06.355 INFO 26087 --- [prometheus-cache-monitor1,63782ef66b0202de,5142754809d54b9c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:30:06.562 INFO 26087 --- [ocp-async-6,00381cf0b5b800f7,7bfd8586e4f63c2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:30:06.601 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:26:06.569, end=2025-04-08T21:28:06.587} 2025-04-08 21:30:06.603 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:30:06.603 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:30:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:30:06.623485+08:00, lastProcessTime=2025-04-08T21:29:56.623279+08:00 2025-04-08 21:30:06.639 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000015, status:SUCCESS 2025-04-08 21:30:06.639 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000015, status:SUCCESS 2025-04-08 21:30:06.662 WARN 26087 --- [ocp-async-5,d4b3a315a50206ac,1b109234d28f15ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:06.662 INFO 26087 --- [ocp-async-7,e3ecb9e1cf9bb1a6,5c0e9f8a4d07eb1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:06.728 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:30:06.623485+08:00 2025-04-08 21:30:06.748 INFO 26087 --- [ocp-schedules-25,d38427ac7c31969c,757b6660691dac38] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:30:08.455 INFO 26087 --- [ocp-schedules-13,95fecc323d8a19dd,a96ea7ca4d0fb65b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:30:08.457 INFO 26087 --- [ocp-schedules-13,95fecc323d8a19dd,a96ea7ca4d0fb65b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:30:08.506 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:08.507 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:08.662 WARN 26087 --- [ocp-async-7,9b580ce081401c40,427925a3e6388d8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:08.662 INFO 26087 --- [ocp-async-2,68e67e3b569878fe,62821ce58dab79c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:09.271 INFO 26087 --- [prometheus-minute-schedule-1,4479869c6bbc6fce,e51467799bae00bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:09.917 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:30:09.918 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:30:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:30:10.579 WARN 26087 --- [ocp-async-6,4c885e6b54e62f12,9551a2fcb01b8032] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:10.579 INFO 26087 --- [ocp-async-8,10febb09b48e24df,417c8e9e63ccde9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:10.995 INFO 26087 --- [ocp-schedules-16,8dad9d4f03bfc707,da925b90016b26e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:30:11.423 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:11.424 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:12.271 INFO 26087 --- [prometheus-minute-schedule-1,a3a6527d7a7ef263,0054e44c9dae5777] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:12.391 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118940, nodeCount=67, affectRows=67 2025-04-08 21:30:12.392 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118940, nodeCount=67, validCount=67 2025-04-08 21:30:12.393 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118940, nodeCount=77, affectRows=77 2025-04-08 21:30:12.393 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118940, nodeCount=80, affectRows=80 2025-04-08 21:30:12.393 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118940, nodeCount=67, affectRows=67 2025-04-08 21:30:12.393 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118940, nodeCount=76, validCount=76 2025-04-08 21:30:12.393 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744118940, nodeCount=87, affectRows=87 2025-04-08 21:30:12.394 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118940, nodeCount=67, validCount=67 2025-04-08 21:30:12.394 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118940, nodeCount=80, validCount=80 2025-04-08 21:30:12.394 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744118940, nodeCount=87, validCount=87 2025-04-08 21:30:12.579 WARN 26087 --- [ocp-async-5,7145f49bff6b26ee,65779773587517c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:12.579 INFO 26087 --- [ocp-async-8,eb09c5d4743b8a58,28ceafc585f41680] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:14.424 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:14.425 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:14.579 WARN 26087 --- [ocp-async-1,f050c1aea961c084,877367eaa05bc7c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:14.579 INFO 26087 --- [ocp-async-4,5b9b43ea1278b8f8,3cfe06021490e681] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:15.271 INFO 26087 --- [prometheus-minute-schedule-1,7c3217bccd524147,e7a969b9f0d0200d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:16.433 INFO 26087 --- [ocp-schedules-15,d3101c463ac44917,98c0834ad9b01e9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:30:16.434 INFO 26087 --- [ocp-schedules-20,b8454ca32536c8d2,d58cecbcbdbb072b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:30:16.435 INFO 26087 --- [ocp-schedules-21,478806a09bd9d3bc,21f2017261ca7f05] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:30:16.435+08:00, tenantSize=0 2025-04-08 21:30:16.437 INFO 26087 --- [ocp-schedules-10,578d115b275bf6b1,1f6c23cda7c87a2a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:30:16.437 INFO 26087 --- [ocp-schedules-10,578d115b275bf6b1,1f6c23cda7c87a2a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:30:16.437 INFO 26087 --- [ocp-schedules-9,b2bac1c746775663,31eb3786413a41cf] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:30:16.439 INFO 26087 --- [ocp-schedules-11,06048a8533c913e4,a313742d6a016d68] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:30:16.439 INFO 26087 --- [ocp-schedules-29,ceed021514e2b297,037b296feb3fe806] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:30:16.439 INFO 26087 --- [ocp-schedules-11,06048a8533c913e4,a313742d6a016d68] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:30:16.439 INFO 26087 --- [ocp-schedules-29,ceed021514e2b297,037b296feb3fe806] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:30:16.441 INFO 26087 --- [ocp-schedules-4,11b7a635b6c19317,c300a913b307050c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:30:16.441 INFO 26087 --- [ocp-schedules-30,e65e8a4e626cd233,064c80a1a4bb329f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:30:16.441 INFO 26087 --- [ocp-schedules-7,a2f19b3c803f8500,c44d875eff800ffb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:30:16.441 INFO 26087 --- [ocp-schedules-20,b8454ca32536c8d2,d58cecbcbdbb072b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:30:16.444 INFO 26087 --- [ocp-schedules-15,d3101c463ac44917,98c0834ad9b01e9b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:30:16.446 INFO 26087 --- [ocp-schedules-19,f167b9367976a059,753b110bfdd4b58f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:30:16.479 INFO 26087 --- [ocp-async-6,d821a6809537488f,cfa61f61586747d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:30:16.523 INFO 26087 --- [ocp-schedules-2,1904462c9337d4ab,3dc49ada568717ef] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:30:16.553 INFO 26087 --- [ocp-schedules-27,619077cdc4da475d,f45111df40ea0295] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:30:16.561 INFO 26087 --- [ocp-schedules-28,020528a18ed89808,30cd178f38da8355] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:30:16.561 INFO 26087 --- [ocp-schedules-28,020528a18ed89808,30cd178f38da8355] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:30:16.572 INFO 26087 --- [ocp-schedules-12,cab60950f1f04655,e1a562536454241a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:30:16.572+08:00, clusters=[] 2025-04-08 21:30:16.574 INFO 26087 --- [ocp-schedules-10,ca7cd8cfc007a674,6cff5d2ec58232db] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:30:16.574 INFO 26087 --- [ocp-schedules-10,ca7cd8cfc007a674,6cff5d2ec58232db] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:30:16.575 INFO 26087 --- [ocp-schedules-21,14466fa3c62acbaf,f0c910b4b8d1ec91] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:30:16.576 INFO 26087 --- [ocp-schedules-16,cee95128aee07871,cd10cbc9ede86bfc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:30:16.576+08:00, clusters=[] 2025-04-08 21:30:16.579 WARN 26087 --- [ocp-async-10,fb8ee82d3b9c2fce,52692fc17eaf64bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:16.579 INFO 26087 --- [ocp-async-5,e3b1fe5bec1be58b,4b6399e1854316d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:16.598 INFO 26087 --- [ocp-schedules-20,53aec66c9f1913bd,020f3cbc74ba66f8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:30:16.598 INFO 26087 --- [ocp-schedules-20,53aec66c9f1913bd,020f3cbc74ba66f8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:30:16.599 INFO 26087 --- [ocp-schedules-20,53aec66c9f1913bd,020f3cbc74ba66f8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:30:16.599 INFO 26087 --- [ocp-schedules-20,53aec66c9f1913bd,020f3cbc74ba66f8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:30:16.603 INFO 26087 --- [ocp-schedules-4,fb0f7df73c0a98de,5ca1a4c95afcad5b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:30:16.604 INFO 26087 --- [ocp-schedules-4,fb0f7df73c0a98de,5ca1a4c95afcad5b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:30:16.604 INFO 26087 --- [ocp-schedules-4,fb0f7df73c0a98de,5ca1a4c95afcad5b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:30:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:30:16.622864+08:00, lastProcessTime=2025-04-08T21:30:06.623485+08:00 2025-04-08 21:30:16.666 INFO 26087 --- [ocp-schedules-15,6bfab5d1a9e1e08d,739f31b72409ac9a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:30:16.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:30:16.622864+08:00 2025-04-08 21:30:16.752 INFO 26087 --- [ocp-schedules-22,3beb1324e0a2fcfa,4824c6f1d8e8b9f6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:30:16.766 INFO 26087 --- [ocp-schedules-24,07d2738de7c960b1,bb37621be3f6fd72] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:30:16.768 INFO 26087 --- [ocp-schedules-18,ce7b5f167167526e,0e15a05f814e44a1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:30:16.794 INFO 26087 --- [ocp-schedules-25,9ac555f2ec1d4106,b8f0464882e6ba44] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:30:16.794 INFO 26087 --- [ocp-schedules-25,9ac555f2ec1d4106,b8f0464882e6ba44] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:30:16.795 INFO 26087 --- [ocp-schedules-25,9ac555f2ec1d4106,b8f0464882e6ba44] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:30:16.795 INFO 26087 --- [ocp-schedules-25,9ac555f2ec1d4106,b8f0464882e6ba44] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:30:16.795 INFO 26087 --- [ocp-schedules-25,9ac555f2ec1d4106,b8f0464882e6ba44] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:30:16.795 INFO 26087 --- [ocp-schedules-25,9ac555f2ec1d4106,b8f0464882e6ba44] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:30:17.425 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:17.425 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:17.495 INFO 26087 --- [ocp-schedules-10,fd36d9a4add07153,bb962fb459628c73] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:30:17.498 INFO 26087 --- [ocp-schedules-16,a168e24edbc361dd,cdd00835824deda9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:30:17.498 INFO 26087 --- [ocp-schedules-16,a168e24edbc361dd,cdd00835824deda9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:30:17.501 INFO 26087 --- [ocp-schedules-10,fd36d9a4add07153,bb962fb459628c73] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:30:17.501 INFO 26087 --- [ocp-schedules-10,fd36d9a4add07153,bb962fb459628c73] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:30:18.271 INFO 26087 --- [prometheus-minute-schedule-1,e09cef4eca3ac041,f8d56c05e70747d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:18.379 INFO 26087 --- [ocp-schedules-13,f79eddebca53b014,00492e73ed86fa44] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:30:18.380 INFO 26087 --- [ocp-schedules-13,f79eddebca53b014,00492e73ed86fa44] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:30:18.579 WARN 26087 --- [ocp-async-5,b9b91742f7f94bce,a8fa3499a05c51dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:18.580 INFO 26087 --- [ocp-async-1,64281c6ae9073994,805c8781545570bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:20.425 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:20.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:30:20.580 WARN 26087 --- [ocp-async-3,d73f7eb0feab8e29,0decd5fb82017e53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:20.580 INFO 26087 --- [ocp-async-4,23ce13808ca663e4,12bfbcf11115009c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:20.996 INFO 26087 --- [ocp-schedules-4,e249aa14c614dca3,f78c82370968d750] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:30:21.271 INFO 26087 --- [prometheus-minute-schedule-1,af122ebb8b8b5c7d,c2ab978d26200e1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:22.580 WARN 26087 --- [ocp-async-2,d2b0f5c35a860b87,0d331bde458d2eff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:22.580 INFO 26087 --- [ocp-async-9,9b1d5e30c9437c27,4e2971b84869644e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:23.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:23.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:24.271 INFO 26087 --- [prometheus-minute-schedule-1,e15a4a4418450e9f,f68f0e902e44f9a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:24.580 WARN 26087 --- [ocp-async-7,210593b857af315b,a24ba65d4cf3bfd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:24.580 INFO 26087 --- [ocp-async-5,b520fdd57aafb781,28201587241b31e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:26.426 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:26.427 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:26.479 INFO 26087 --- [ocp-async-1,c8592a2d4924f223,421c7218ece82f38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:30:26.580 WARN 26087 --- [ocp-async-4,aae6f2c1d12f0dfb,69c0c70d0fbe9e36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:26.580 INFO 26087 --- [ocp-async-6,f6eda8f3a9f6ac84,062d6d36a85fc014] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:30:26.623216+08:00, lastProcessTime=2025-04-08T21:30:16.622864+08:00 2025-04-08 21:30:26.667 INFO 26087 --- [ocp-schedules-7,a39f0aab017d7ce0,4a0dbd42563daf0e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:30:26.713 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:30:26.623216+08:00 2025-04-08 21:30:27.271 INFO 26087 --- [prometheus-minute-schedule-1,c7b7514378a98819,9609388079cf31d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:27.443 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:30:27.443 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:30:27.446 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:30:27.446 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:30:27.446 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:30:27.446 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:30:27.450 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:30:27.450 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:30:27.450 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:30:27.450 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:30:27.472 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:30:27.477 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:30:27.477 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:30:27.477 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:30:27.477 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:30:27.477 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:30:27.482 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:30:27.795 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:30:28.111 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.661301538S 2025-04-08 21:30:28.112 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:30:28.168 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:30:28.168 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:30:28.192 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:30:28.227 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:30:28.227 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:30:28.227 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:30:28.227 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:30:28.227 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:30:28.227 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:30:28.234 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:30:28.331 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] o.s.boot.SpringApplication : Started application in 0.885 seconds (JVM running for 1733.788) 2025-04-08 21:30:28.331 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.885186234S 2025-04-08 21:30:28.385 INFO 26087 --- [ocp-schedules-4,fbe416e598db96e4,16e516ded556a42a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:30:28.386 INFO 26087 --- [ocp-schedules-4,fbe416e598db96e4,16e516ded556a42a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:30:28.580 WARN 26087 --- [ocp-async-10,426212683fccb927,4dcc3a4cf0cb55c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:28.580 INFO 26087 --- [ocp-async-9,940b5e08c31b901f,67036636f03d0980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:28.606 INFO 26087 --- [ocp-schedules-18,f6aa0b5fb94dcd30,a8b93a424e14f28b] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:30:29.427 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:29.428 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:30.271 INFO 26087 --- [prometheus-minute-schedule-1,0f7126f9ce2c53a1,a72077bce15292cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:30.580 WARN 26087 --- [ocp-async-8,51dc40ac534f27f6,96bc9a69cf5d9bd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:30.580 INFO 26087 --- [ocp-async-1,24276bac7b7ab61c,def3cd8cef90b18a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:31.023 INFO 26087 --- [ocp-schedules-6,0d8a48f9f4ef30d4,7e59f0ce6113ea24] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:30:32.428 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:32.428 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:32.580 WARN 26087 --- [ocp-async-3,1fac1332ad741844,043b4d332bf38ae8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:32.581 INFO 26087 --- [ocp-async-4,763bf8ddda24a1a1,06cb9c68f7a1737c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:33.271 INFO 26087 --- [prometheus-minute-schedule-1,f4617f34e6d27e7c,86c6f166fe558009] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:34.581 WARN 26087 --- [ocp-async-6,c06ecccd4dafd07c,06ad9f80bcacd688] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:34.581 INFO 26087 --- [ocp-async-7,8de0e2d38f8cc3ff,f810295f9d24756b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:35.429 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:35.429 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:36.271 INFO 26087 --- [prometheus-minute-schedule-1,b33ff46a4b2c9d30,d3b30c1f09ea0da3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:36.479 INFO 26087 --- [ocp-async-7,3ef6a187d6dc9fdb,0d210c1f8e9e31fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:30:36.581 WARN 26087 --- [ocp-async-8,d3b79a29629c3b90,1355aedc2a04902e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:36.581 INFO 26087 --- [ocp-async-5,05f6498c7013092a,6c779e9430c266c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:30:36.622634+08:00, lastProcessTime=2025-04-08T21:30:26.623216+08:00 2025-04-08 21:30:36.668 INFO 26087 --- [ocp-schedules-27,a05b15c3596bc5be,2baa0bbd99c01e75] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:30:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:30:36.622634+08:00 2025-04-08 21:30:37.502 INFO 26087 --- [ocp-schedules-26,88fc950e84393065,e9914a47fa9d8754] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:30:37.517 INFO 26087 --- [ocp-schedules-26,88fc950e84393065,e9914a47fa9d8754] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:30:37.517 INFO 26087 --- [ocp-schedules-26,88fc950e84393065,e9914a47fa9d8754] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:30:38.429 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:38.430 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:38.431 INFO 26087 --- [ocp-schedules-4,3e1c9bf0b80b96f4,782cabcbfa3872c1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:30:38.432 INFO 26087 --- [ocp-schedules-4,3e1c9bf0b80b96f4,782cabcbfa3872c1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:30:38.581 WARN 26087 --- [ocp-async-4,f858548e51bebc58,ba10f980f0d08477] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:38.581 INFO 26087 --- [ocp-async-3,3c3d5bd370fa8c3a,0c02ed4db576b459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:39.271 INFO 26087 --- [prometheus-minute-schedule-1,b0b38f6ab3a9028d,7c963a92aa0deb96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:40.001 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:30:40.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:30:40.581 WARN 26087 --- [ocp-async-7,63f7ef3197bdefd1,df430991ef888ca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:40.581 INFO 26087 --- [ocp-async-6,e1f4cf9c61d99fb3,e7be19795030048f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:41.041 INFO 26087 --- [ocp-schedules-3,4f4c8ca894071eb9,e841d64fea0d2850] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:30:41.430 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:41.430 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:42.271 INFO 26087 --- [prometheus-minute-schedule-1,0c123cb0bca9ce6b,6cca13783257399d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:42.581 WARN 26087 --- [ocp-async-1,5de20bb5eae20d97,f986ba7e5505ad34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:42.581 INFO 26087 --- [ocp-async-8,b1ec4e098410c75f,abfd3bf3d3556f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:44.430 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:44.431 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:44.581 INFO 26087 --- [ocp-async-2,6c0ccacd432252b3,82944fb48c155761] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:44.581 WARN 26087 --- [ocp-async-5,304343d1392f0ef7,15210980015b10e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:45.271 INFO 26087 --- [prometheus-minute-schedule-1,73269fe607d41007,fec9071fb69ae2cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:46.437 INFO 26087 --- [ocp-schedules-1,3cfc31afaaaad4f6,2cc6a6173358a6bd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:30:46.437 INFO 26087 --- [ocp-schedules-27,5cf1a5ea70fba8d1,f0a223f0509d740d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:30:46.437 INFO 26087 --- [ocp-schedules-27,5cf1a5ea70fba8d1,f0a223f0509d740d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:30:46.440 INFO 26087 --- [ocp-schedules-4,1468d4b4557937cd,ae4e0bafda0842db] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:30:46.440 INFO 26087 --- [ocp-schedules-4,1468d4b4557937cd,ae4e0bafda0842db] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:30:46.440 INFO 26087 --- [ocp-schedules-9,23705077a0547eba,e0bce364a643aa76] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:30:46.440 INFO 26087 --- [ocp-schedules-9,23705077a0547eba,e0bce364a643aa76] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:30:46.441 INFO 26087 --- [ocp-schedules-26,086554b9f0dee221,58d0bce558ccbfba] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:30:46.441 INFO 26087 --- [ocp-schedules-24,09d0b591a8f5f272,1c9c636ea6873cab] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:30:46.441 INFO 26087 --- [ocp-schedules-16,50e4e4d622f74a84,5fcb2f28cd428018] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:30:46.442 INFO 26087 --- [ocp-schedules-15,d285d41d3420bc94,4ad236fc542bac51] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:30:46.480 INFO 26087 --- [ocp-async-2,0f52a052245c637a,9d46c6df224c02b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:30:46.554 INFO 26087 --- [ocp-schedules-29,bd5b819a91f6b014,dceae9a096fe91d5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:30:46.581 INFO 26087 --- [ocp-async-10,7d7b7efbf35d6b8c,d4660f747402a77d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:46.582 WARN 26087 --- [ocp-async-6,3158f9355b9c8c69,7440a72ccc3130c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:30:46.623355+08:00, lastProcessTime=2025-04-08T21:30:36.622634+08:00 2025-04-08 21:30:46.670 INFO 26087 --- [ocp-schedules-17,a6c1b930a6d7d543,33b4c757ae797021] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:30:46.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:30:46.623355+08:00 2025-04-08 21:30:47.431 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:47.432 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:48.271 INFO 26087 --- [prometheus-minute-schedule-1,cc60187641f75ef2,ea50da4671ffad39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:48.441 INFO 26087 --- [ocp-schedules-17,2ab3fc735136d5d1,48a6b635fa8eb437] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:30:48.443 INFO 26087 --- [ocp-schedules-17,2ab3fc735136d5d1,48a6b635fa8eb437] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:30:48.582 WARN 26087 --- [ocp-async-8,938d1a4100346bb7,cc676e5124eb2da0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:48.582 INFO 26087 --- [ocp-async-1,244953d44afa8005,a38b612aaaea8bc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:50.432 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:50.433 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:50.582 WARN 26087 --- [ocp-async-4,047e67e6ebe83d23,7dde227ac6b5386f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:50.582 INFO 26087 --- [ocp-async-5,12e60c9572a2b706,c4c6d72eb48992f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:51.061 INFO 26087 --- [ocp-schedules-27,7b4e8d0639596d49,ac60dff582187da8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:30:51.271 INFO 26087 --- [prometheus-minute-schedule-1,e71877646fe44e57,6265af4febbc3e8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:52.582 WARN 26087 --- [ocp-async-3,76f1d2ef7255198c,f91550b3e8add570] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:52.582 INFO 26087 --- [ocp-async-7,589c0e5cff1643eb,b644323d160b04c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:53.433 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:53.433 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:54.271 INFO 26087 --- [prometheus-minute-schedule-1,e87dcbab577a197a,8087ece6b52b64b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:54.582 WARN 26087 --- [ocp-async-9,9f738e11e94d107e,49396a749cd301fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:54.582 INFO 26087 --- [ocp-async-8,aaf1ea033243eb62,08b6fccb16357cdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:56.433 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:56.434 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:30:56.480 INFO 26087 --- [ocp-async-10,49b5c511a259748e,d99021d73c8c1413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:30:56.495 INFO 26087 --- [ocp-schedules-13,23b2bd3dc9020e88,6d9b40f1d0830e02] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:30:56.582 INFO 26087 --- [ocp-async-4,aae6d934c0696795,6d6a7ac7ebc90aa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:56.582 WARN 26087 --- [ocp-async-2,7993608140b68818,b3af2ba871e5ddd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:30:56.623653+08:00, lastProcessTime=2025-04-08T21:30:46.623355+08:00 2025-04-08 21:30:56.671 INFO 26087 --- [ocp-schedules-21,44580db921930391,c3b32c713a6cfd98] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:30:56.745 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:30:56.623653+08:00 2025-04-08 21:30:57.271 INFO 26087 --- [prometheus-minute-schedule-1,4902f11629cf9185,261c74e7f9a91351] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119060, interval=60 2025-04-08 21:30:57.519 INFO 26087 --- [ocp-schedules-7,4bc14441d4a651a3,dacc16942a5b7651] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:30:57.529 INFO 26087 --- [ocp-schedules-7,4bc14441d4a651a3,dacc16942a5b7651] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:30:57.529 INFO 26087 --- [ocp-schedules-7,4bc14441d4a651a3,dacc16942a5b7651] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:30:58.446 INFO 26087 --- [ocp-schedules-17,96a52160427c4c02,6d7406d2de47c1b9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:30:58.447 INFO 26087 --- [ocp-schedules-17,96a52160427c4c02,6d7406d2de47c1b9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:30:58.582 WARN 26087 --- [ocp-async-6,ce217d16b560b580,e90a355681e72025] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:30:58.582 INFO 26087 --- [ocp-async-3,943c78d5d8d6ddcf,65b6348589638aeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:30:59.434 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:30:59.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:00.271 INFO 26087 --- [prometheus-minute-schedule-1,cb6c6d50a28b9a68,0bb8c21223750bd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:00.582 WARN 26087 --- [ocp-async-9,fe7f3fa01dd634d4,6d0220cff971a5c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:00.582 INFO 26087 --- [ocp-async-3,6c2afd31b9af34d7,84cffcdcc5ff9514] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:01.078 INFO 26087 --- [ocp-schedules-16,0698758f2fbda7b4,fe89b8f2d25f2711] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:31:02.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:02.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:02.583 WARN 26087 --- [ocp-async-3,63b4e2849a81df2c,2e7f21adf54f9d54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:02.583 INFO 26087 --- [ocp-async-2,3962e961cf493249,830d51be71ac1a3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:03.271 INFO 26087 --- [prometheus-minute-schedule-1,1877432788d18622,84623649ade22fa8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:04.583 WARN 26087 --- [ocp-async-6,0a4523ca686e121c,7b13926c52ba5334] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:04.583 INFO 26087 --- [ocp-async-6,616df95c6720334b,5e0ce860ac8cd5d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:05.435 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:05.436 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:06.271 INFO 26087 --- [prometheus-minute-schedule-1,bf6ffefee8a7ab84,8c7516bc151a0570] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:06.272 INFO 26087 --- [prometheus-cache-monitor1,1ba16b1d25a1deac,94c6c70a4755fa73] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:31:06.272 INFO 26087 --- [prometheus-cache-monitor1,1ba16b1d25a1deac,94c6c70a4755fa73] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:31:06.480 INFO 26087 --- [ocp-async-6,bb3fbb81098cdb93,2784773a2bd0d7ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:31:06.583 WARN 26087 --- [ocp-async-9,8ddf259b4f1b0647,a02dc27c8a7823dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:06.583 INFO 26087 --- [ocp-async-8,e15f0ad2ae5df3e6,8a6e19129c9b2c9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:31:06.623018+08:00, lastProcessTime=2025-04-08T21:30:56.623653+08:00 2025-04-08 21:31:06.673 INFO 26087 --- [ocp-schedules-23,e7acb9f3fb014432,4be927e53d9fad3d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:31:06.717 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:31:06.623018+08:00 2025-04-08 21:31:08.436 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:08.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:08.452 INFO 26087 --- [ocp-schedules-27,a384c2dd17081188,068c5339adaace5b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:31:08.453 INFO 26087 --- [ocp-schedules-27,a384c2dd17081188,068c5339adaace5b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:31:08.583 WARN 26087 --- [ocp-async-5,df1feeb5a3a8eca6,02390d7fffba21c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:08.583 INFO 26087 --- [ocp-async-1,da8a65915ed2f109,fc2487ea46bd7565] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:09.271 INFO 26087 --- [prometheus-minute-schedule-1,1446c22cc5ba0bcf,c56c9012981be902] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:10.002 INFO 26087 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:31:10.002 INFO 26087 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:31:10.521 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:31:10.583 WARN 26087 --- [ocp-async-4,cead048a60f8971f,57cabbd47d289972] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:10.583 INFO 26087 --- [ocp-async-2,2d766af371da6b91,2295a67089a4070b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:11.099 INFO 26087 --- [ocp-schedules-7,588d1d933ab9c150,f8575102c8a9f7c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:31:11.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:11.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:12.271 INFO 26087 --- [prometheus-minute-schedule-1,fcdb51293428e7b7,c50b3677662022fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:12.420 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119000, nodeCount=378, affectRows=378 2025-04-08 21:31:12.422 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119000, nodeCount=377, validCount=377 2025-04-08 21:31:12.583 WARN 26087 --- [ocp-async-10,339487791441d2d6,fedbc3fbd81dc7d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:12.583 INFO 26087 --- [ocp-async-2,401d9e58a760cd5d,985961c3280e3fa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:14.437 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:14.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:14.584 WARN 26087 --- [ocp-async-8,8922444c575afbf1,a4294ba4728d4609] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:14.584 INFO 26087 --- [ocp-async-2,b0c849c84b99d218,a26b0f3b684d652e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:15.271 INFO 26087 --- [prometheus-minute-schedule-1,7d20b1a1c85e5825,4a3b38cce36a8e83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:16.433 INFO 26087 --- [ocp-schedules-17,bb59390011594c88,6fb4b15ab31356c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:31:16.434 INFO 26087 --- [ocp-schedules-13,7de54c7a004d96a0,54c1c9790820ab00] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:31:16.437 INFO 26087 --- [ocp-schedules-25,cb292376672aa081,bfc042ce93077892] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:31:16.437 INFO 26087 --- [ocp-schedules-23,71de06f25dd61c9d,978b66a795f7439a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:31:16.437 INFO 26087 --- [ocp-schedules-25,cb292376672aa081,bfc042ce93077892] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:31:16.439 INFO 26087 --- [ocp-schedules-22,68315044cc95ff80,06685a03b00bba84] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:31:16.439 INFO 26087 --- [ocp-schedules-22,68315044cc95ff80,06685a03b00bba84] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:31:16.439 INFO 26087 --- [ocp-schedules-16,fa1acd561faff760,ebefadfef86b0c68] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:31:16.440 INFO 26087 --- [ocp-schedules-16,fa1acd561faff760,ebefadfef86b0c68] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:31:16.441 INFO 26087 --- [ocp-schedules-13,7de54c7a004d96a0,54c1c9790820ab00] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:31:16.441 INFO 26087 --- [ocp-schedules-24,811c95c7a71046b6,ae1bbd52098fd8dc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:31:16.441 INFO 26087 --- [ocp-schedules-10,6daacb534e3742f1,fab7e35ac91bf74c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:31:16.441 INFO 26087 --- [ocp-schedules-29,a598ef0fed905094,2ebdff39b5368d3d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:31:16.465 INFO 26087 --- [ocp-schedules-17,bb59390011594c88,6fb4b15ab31356c0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:31:16.480 INFO 26087 --- [ocp-schedules-1,28c7ca910a33e60a,6e3c0d9154e4f6c7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:31:16.480 INFO 26087 --- [ocp-schedules-1,28c7ca910a33e60a,6e3c0d9154e4f6c7] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:31:16.480 INFO 26087 --- [ocp-async-4,39216862a10f98ba,5c4aded51f60236a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:31:16.503 INFO 26087 --- [ocp-schedules-30,d58d96c8da1ac31b,6be0690bf88cd47a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:31:16.503 INFO 26087 --- [ocp-schedules-30,d58d96c8da1ac31b,6be0690bf88cd47a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:31:16.524 INFO 26087 --- [ocp-schedules-12,41fcaebbe93c8a93,3f53216d27935208] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:31:16.534 INFO 26087 --- [ocp-schedules-25,211a2ae65c455833,d8a0e0ef9c26c56b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:31:16.555 INFO 26087 --- [ocp-schedules-28,9919688ce9368744,c19a07e014f8bfed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:31:16.563 INFO 26087 --- [ocp-schedules-27,032deb5e3ac92cd4,5ca40c1d90c9196b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:31:16.563 INFO 26087 --- [ocp-schedules-27,032deb5e3ac92cd4,5ca40c1d90c9196b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:31:16.574 INFO 26087 --- [ocp-schedules-3,9421c94fe3befe2a,5cf3f5ed791546c7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:31:16.574+08:00, clusters=[] 2025-04-08 21:31:16.575 INFO 26087 --- [ocp-schedules-7,cc019d59619f98f1,799202f1fbd65390] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:31:16.575 INFO 26087 --- [ocp-schedules-7,cc019d59619f98f1,799202f1fbd65390] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:31:16.576 INFO 26087 --- [ocp-schedules-5,648c821cb37cb9cf,905e58ee8ee5ebb1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:31:16.577 INFO 26087 --- [ocp-schedules-23,52f7b24760c274d8,19ad749058173470] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:31:16.577+08:00, clusters=[] 2025-04-08 21:31:16.584 WARN 26087 --- [ocp-async-9,66d904a49fef914a,de52d959d8113087] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:16.584 INFO 26087 --- [ocp-async-10,a2b96a06cb36511b,9257ef46c00ea1d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:16.600 INFO 26087 --- [ocp-schedules-22,9658088ad890e7d0,792c7e34ee244290] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:31:16.600 INFO 26087 --- [ocp-schedules-22,9658088ad890e7d0,792c7e34ee244290] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:31:16.601 INFO 26087 --- [ocp-schedules-22,9658088ad890e7d0,792c7e34ee244290] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:31:16.601 INFO 26087 --- [ocp-schedules-22,9658088ad890e7d0,792c7e34ee244290] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:31:16.604 INFO 26087 --- [ocp-schedules-24,f43f16ebbce1af9d,85bf4587f3daeff3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:31:16.605 INFO 26087 --- [ocp-schedules-24,f43f16ebbce1af9d,85bf4587f3daeff3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:31:16.605 INFO 26087 --- [ocp-schedules-24,f43f16ebbce1af9d,85bf4587f3daeff3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:31:16.622 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:31:16.622630+08:00, lastProcessTime=2025-04-08T21:31:06.623018+08:00 2025-04-08 21:31:16.674 INFO 26087 --- [ocp-schedules-11,02287c8bd87af1fc,3fd00526badd0b62] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:31:16.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:31:16.622630+08:00 2025-04-08 21:31:16.751 INFO 26087 --- [ocp-schedules-1,d853b9f3758e041b,f7001d60050cb639] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:31:16.753 INFO 26087 --- [ocp-schedules-21,02067d21c1e6b5f1,7d04eb7c6e9ce406] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:31:16.766 INFO 26087 --- [ocp-schedules-2,42a0a53a56ed9bc3,ddcf821b94c55055] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:31:16.771 INFO 26087 --- [ocp-schedules-14,19bd1681dea6ac7c,e7d1b084a69b4cc5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:31:16.795 INFO 26087 --- [ocp-schedules-19,5325a01114abba91,c719639f76e3e311] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:31:16.795 INFO 26087 --- [ocp-schedules-19,5325a01114abba91,c719639f76e3e311] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:31:16.798 INFO 26087 --- [ocp-schedules-19,5325a01114abba91,c719639f76e3e311] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:31:16.798 INFO 26087 --- [ocp-schedules-19,5325a01114abba91,c719639f76e3e311] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:31:16.801 INFO 26087 --- [ocp-schedules-19,5325a01114abba91,c719639f76e3e311] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:31:16.801 INFO 26087 --- [ocp-schedules-19,5325a01114abba91,c719639f76e3e311] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:31:17.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:17.438 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:17.501 INFO 26087 --- [ocp-schedules-7,165f4b8bc0d7271a,5d8cb1f63c410b45] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:31:17.501 INFO 26087 --- [ocp-schedules-7,165f4b8bc0d7271a,5d8cb1f63c410b45] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:31:17.530 INFO 26087 --- [ocp-schedules-13,36eec19be8fa98e6,8b12843b6640d50d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:31:17.542 INFO 26087 --- [ocp-schedules-13,36eec19be8fa98e6,8b12843b6640d50d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:31:17.542 INFO 26087 --- [ocp-schedules-13,36eec19be8fa98e6,8b12843b6640d50d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:31:18.271 INFO 26087 --- [prometheus-minute-schedule-1,173d75fdd9f99064,30e28b68f20abfc3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:18.461 INFO 26087 --- [ocp-schedules-20,118856dc2cad60d6,0a1e79128957da4a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:31:18.463 INFO 26087 --- [ocp-schedules-20,118856dc2cad60d6,0a1e79128957da4a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:31:18.584 WARN 26087 --- [ocp-async-7,133a2c888bd19d5d,b41e636f631187d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:18.584 INFO 26087 --- [ocp-async-10,910401e47bef2327,c8c828b37bc959d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:20.439 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:20.439 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:31:20.584 WARN 26087 --- [ocp-async-10,99e4b97b0f5934b8,8a6ade7f511909c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:20.584 INFO 26087 --- [ocp-async-1,8b2a540466f7979d,177392c8dc1c841e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:21.116 INFO 26087 --- [ocp-schedules-4,3da8361c28567e4b,5896d386472b9d8f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:31:21.271 INFO 26087 --- [prometheus-minute-schedule-1,ca11d5dbfedbedf0,c19f9f31332d463d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:22.584 WARN 26087 --- [ocp-async-1,a9c1ba2251a4c120,027833d7ccc370b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:22.584 INFO 26087 --- [ocp-async-9,4aca29901336643e,d7d15814b666c4e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:23.439 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:23.440 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:24.271 INFO 26087 --- [prometheus-minute-schedule-1,a41bcc10d3fe8cd7,34978da20855de04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:24.584 WARN 26087 --- [ocp-async-8,614f96310dea294f,afe7fd954d8365c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:24.584 INFO 26087 --- [ocp-async-6,bc7fc30ae9972060,35eebe9df6eaa2f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:26.440 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:26.440 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:26.481 INFO 26087 --- [ocp-async-2,5b21285d1aa27d3a,d3414d210f4bba25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:31:26.584 WARN 26087 --- [ocp-async-5,426a8a1f88fd980f,0ab1c7588a7eb95f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:26.584 INFO 26087 --- [ocp-async-1,db5c4a803d3ac09f,69857aadbb1064ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:26.626 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:31:26.625827+08:00, lastProcessTime=2025-04-08T21:31:16.622630+08:00 2025-04-08 21:31:26.675 INFO 26087 --- [ocp-schedules-30,f5a8911c3ec1c173,b83debf6fe79da91] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:31:26.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:31:26.625827+08:00 2025-04-08 21:31:27.271 INFO 26087 --- [prometheus-minute-schedule-1,836fe219fc96eb36,8de6d64c5b9b531a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:28.468 INFO 26087 --- [ocp-schedules-19,585fabac7cad3ebc,e392300eb89a318e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:31:28.469 INFO 26087 --- [ocp-schedules-19,585fabac7cad3ebc,e392300eb89a318e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:31:28.585 WARN 26087 --- [ocp-async-3,5ce48130800b3489,cad376a8a61f3fb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:28.585 INFO 26087 --- [ocp-async-9,540f950e97073b8f,17bf8d5d48fe9354] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:29.441 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:29.441 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:30.271 INFO 26087 --- [prometheus-minute-schedule-1,44046b8de76e6735,6dd2445a5c766c52] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:30.585 INFO 26087 --- [ocp-async-6,20ebf14b73b4c66e,b59a4ba3cdb5c4e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:30.585 WARN 26087 --- [ocp-async-2,e3c1c882c557a35a,f54ec746fca54b5b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:31.136 INFO 26087 --- [ocp-schedules-16,176d120194315bc6,8af3dd44b72da145] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:31:32.441 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:32.442 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:32.585 WARN 26087 --- [ocp-async-4,ae0cf2fb5d4a8294,626c0948296ae1e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:32.585 INFO 26087 --- [ocp-async-5,d1c8ca341100d4cc,6b2c0ad17b713dcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:33.271 INFO 26087 --- [prometheus-minute-schedule-1,81ba4a7cbdcb6549,6c03f32d1de76cc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:34.585 INFO 26087 --- [ocp-async-3,0a644f41ed31a253,44be458db19497ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:34.585 WARN 26087 --- [ocp-async-8,33187c9910007503,919b56cb9ec43f1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:34.909 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:31:34.909 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:31:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:31:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:31:34.911 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:31:34.912 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:31:35.442 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:35.443 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:36.271 INFO 26087 --- [prometheus-minute-schedule-1,b16a4f5d913df6f1,0b15d821da0d0a10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:36.481 INFO 26087 --- [ocp-async-3,ee6dcaa6e9359117,dedf3288b76e6a70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:31:36.585 INFO 26087 --- [ocp-async-2,fa9a4eb31d7fbfd5,08837fe47147c7f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:36.585 WARN 26087 --- [ocp-async-10,97b2f519f288aa1d,528fcceb368af8d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:31:36.623549+08:00, lastProcessTime=2025-04-08T21:31:26.625827+08:00 2025-04-08 21:31:36.677 INFO 26087 --- [ocp-schedules-22,a8512c18f35d7b7a,1ca2bbcf24c3c887] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:31:36.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:31:36.623549+08:00 2025-04-08 21:31:37.543 INFO 26087 --- [ocp-schedules-30,acea67a5a6f7469f,a76fbf94ab4c539d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:31:37.561 INFO 26087 --- [ocp-schedules-30,acea67a5a6f7469f,a76fbf94ab4c539d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:31:37.561 INFO 26087 --- [ocp-schedules-30,acea67a5a6f7469f,a76fbf94ab4c539d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:31:38.443 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:38.443 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:38.475 INFO 26087 --- [ocp-schedules-11,e366c607173921f2,2e25edc3346b4582] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:31:38.476 INFO 26087 --- [ocp-schedules-11,e366c607173921f2,2e25edc3346b4582] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:31:38.585 WARN 26087 --- [ocp-async-5,c5eab4172eaf1ea9,6cce155fd4c6126a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:38.585 INFO 26087 --- [ocp-async-4,e87ac79f6acf8140,1be39f13915923b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:39.271 INFO 26087 --- [prometheus-minute-schedule-1,fba73bfa1fa54383,3e00f845778f9eee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:40.001 INFO 26087 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:31:40.001 INFO 26087 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:31:40.585 WARN 26087 --- [ocp-async-1,55c74bfbdb813614,cb98ba607a52e20d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:40.586 INFO 26087 --- [ocp-async-3,98e709c345a273a5,630ddab78da1a59f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:41.166 INFO 26087 --- [ocp-schedules-11,d2348fc2dc579708,6588d568c736d3f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:31:41.443 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:41.444 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:42.271 INFO 26087 --- [prometheus-minute-schedule-1,0efb49d1aeeb7316,2a75e7160eed3a71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:42.586 WARN 26087 --- [ocp-async-9,7717d2edfb47205f,3015c57b7e4a4d61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:42.586 INFO 26087 --- [ocp-async-10,9f00518044c07bd0,a1e8857aec3d3fe9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:44.444 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:44.444 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:44.586 WARN 26087 --- [ocp-async-2,844a09ac22fca45d,042da9c201fa64aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:44.586 INFO 26087 --- [ocp-async-6,c9a9fa6f89306d7d,7e5ae758bd9b5bdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:45.271 INFO 26087 --- [prometheus-minute-schedule-1,bc1b6444ca47c6a2,3da782a5cb4cf372] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:46.437 INFO 26087 --- [ocp-schedules-26,23825716249be7fd,4292936c80fd60a6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:31:46.438 INFO 26087 --- [ocp-schedules-21,8cb5f997dbbd5b85,1f618544a248ebc6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:31:46.438 INFO 26087 --- [ocp-schedules-21,8cb5f997dbbd5b85,1f618544a248ebc6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:31:46.439 INFO 26087 --- [ocp-schedules-10,d77991a9a79c9664,194edf8849d99e22] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:31:46.439 INFO 26087 --- [ocp-schedules-10,d77991a9a79c9664,194edf8849d99e22] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:31:46.440 INFO 26087 --- [ocp-schedules-20,e7cd5892dc54b497,42f1069141133f29] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:31:46.440 INFO 26087 --- [ocp-schedules-20,e7cd5892dc54b497,42f1069141133f29] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:31:46.441 INFO 26087 --- [ocp-schedules-18,3fcbd08f89b16aad,72a6504ca75d14b9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:31:46.441 INFO 26087 --- [ocp-schedules-29,8e558174e2c18bd6,31567d540566ba3f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:31:46.441 INFO 26087 --- [ocp-schedules-28,e69a3afce89bf700,0ebc4eb4317c9193] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:31:46.441 INFO 26087 --- [ocp-schedules-11,0006ca4b8125a928,4c6de466590ad73f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:31:46.481 INFO 26087 --- [ocp-async-6,df639b8149f35c54,433187a7aae587a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:31:46.557 INFO 26087 --- [ocp-schedules-5,46d701d563e1e551,b03d950f3f1d348a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:31:46.586 WARN 26087 --- [ocp-async-8,0b711a3690c3559c,3370ef9244304463] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:46.586 INFO 26087 --- [ocp-async-3,5f9464ba89f13a1f,c9129e2b581b9eb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:31:46.622996+08:00, lastProcessTime=2025-04-08T21:31:36.623549+08:00 2025-04-08 21:31:46.678 INFO 26087 --- [ocp-schedules-30,391c4fe1207d4e66,d453b123f678a6cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:31:46.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:31:46.622996+08:00 2025-04-08 21:31:47.445 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:47.445 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:48.271 INFO 26087 --- [prometheus-minute-schedule-1,aabc1323f30c11f0,94ca9985be118bdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:48.481 INFO 26087 --- [ocp-schedules-9,7ad42b0a1aad2735,f72230c15c991030] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:31:48.482 INFO 26087 --- [ocp-schedules-9,7ad42b0a1aad2735,f72230c15c991030] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:31:48.586 WARN 26087 --- [ocp-async-7,3bb67d375ad3b50a,3c90f0c53c03150c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:48.586 INFO 26087 --- [ocp-async-10,cea381bb415bc964,d92ea9b2408312f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:50.445 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:50.445 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:50.586 WARN 26087 --- [ocp-async-5,8bf51ae5468d3c00,2efca4f1b2930d63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:50.586 INFO 26087 --- [ocp-async-6,fe59b080eb18c232,d41123f8b7ee1757] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:51.190 INFO 26087 --- [ocp-schedules-21,84bdf688f744b8bc,1949af8e5a9cc7d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:31:51.271 INFO 26087 --- [prometheus-minute-schedule-1,b3714ff2ed8d2c09,bc01640ba68b6608] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:52.586 WARN 26087 --- [ocp-async-4,81fc0a8ac5e0d72c,85c6f083c76bddcc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:52.586 INFO 26087 --- [ocp-async-8,cf2b0eb55302f072,667136ea169a9449] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:53.445 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:53.446 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:54.271 INFO 26087 --- [prometheus-minute-schedule-1,6bfaa8c7fe6d4732,8d6501f00504d845] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:54.586 WARN 26087 --- [ocp-async-3,65c0fb2c37aeaef5,997166257b52502e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:54.586 INFO 26087 --- [ocp-async-9,7a14e34623dcc27f,64979cae1d38067b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:56.446 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:56.447 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:31:56.481 INFO 26087 --- [ocp-async-7,714b05ec7f86f45d,fd2cc8a210265602] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:31:56.495 INFO 26087 --- [ocp-schedules-3,eeb17b4dec6daa11,0cab66104531177e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:31:56.587 WARN 26087 --- [ocp-async-5,fb5e4099f0d94642,a7d3aca66c523c7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:56.587 INFO 26087 --- [ocp-async-6,cc9c7c1304ab2267,2cb1801d73053f69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:56.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:31:56.624885+08:00, lastProcessTime=2025-04-08T21:31:46.622996+08:00 2025-04-08 21:31:56.679 INFO 26087 --- [ocp-schedules-29,1907aa36fa02a382,b95242e2c8149f7b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:31:56.721 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:31:56.624885+08:00 2025-04-08 21:31:57.271 INFO 26087 --- [prometheus-minute-schedule-1,86ce1c9bdc153a73,854c9bf64a7ca9c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119120, interval=60 2025-04-08 21:31:57.562 INFO 26087 --- [ocp-schedules-1,41d799d4727e93d3,f419ef6fb27050a2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:31:57.588 INFO 26087 --- [ocp-schedules-1,41d799d4727e93d3,f419ef6fb27050a2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:31:57.588 INFO 26087 --- [ocp-schedules-1,41d799d4727e93d3,f419ef6fb27050a2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:31:58.488 INFO 26087 --- [ocp-schedules-11,ead943d1a036699a,300202ca13033d9d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:31:58.489 INFO 26087 --- [ocp-schedules-11,ead943d1a036699a,300202ca13033d9d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:31:58.587 WARN 26087 --- [ocp-async-6,9b12c484094c24b4,caf6f7f1080929c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:31:58.587 INFO 26087 --- [ocp-async-1,1229cd83e30cd323,fcc86eca059b5213] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:31:59.447 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:31:59.447 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:00.271 INFO 26087 --- [prometheus-minute-schedule-1,73e48278f5852cf6,b0c56868d51c2c9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:00.587 WARN 26087 --- [ocp-async-1,7f395084314fcf35,7826fad35702ac79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:00.587 INFO 26087 --- [ocp-async-3,04f7d16146a744c8,eb33d499b50a4f43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:01.216 INFO 26087 --- [ocp-schedules-23,78efb2bd12d2099b,702e6b945ae4681e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:32:02.447 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:02.448 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:02.587 WARN 26087 --- [ocp-async-10,289ce2d52919d862,c9b8ccd9ad9887b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:02.587 INFO 26087 --- [ocp-async-3,5801f4a58687e018,16f0917ee884cd5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:03.271 INFO 26087 --- [prometheus-minute-schedule-1,7f1ed965e049a298,e1bd5867d736f8dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:04.587 WARN 26087 --- [ocp-async-6,834d7e477c7d5f94,30197e3624db06a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:04.587 INFO 26087 --- [ocp-async-3,3b24495dbc689966,452d518673276a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:05.448 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:05.449 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:06.271 INFO 26087 --- [prometheus-minute-schedule-1,7a6db6e84949e721,ba5dcea2aa117fb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:06.272 INFO 26087 --- [prometheus-cache-monitor1,d028cbe1a9f5c473,4fe0606ee6d78dbf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:32:06.272 INFO 26087 --- [prometheus-cache-monitor1,d028cbe1a9f5c473,4fe0606ee6d78dbf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:32:06.481 INFO 26087 --- [ocp-async-3,13b2cc50dff8ece2,d73bb077e2e1d071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:32:06.537 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:28:06.587, end=2025-04-08T21:30:06.533} 2025-04-08 21:32:06.542 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:32:06.542 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:32:06.587 WARN 26087 --- [ocp-async-9,2e518949db73b62e,a19ad2b9b1771f5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:06.587 INFO 26087 --- [ocp-async-1,6c74265aac723154,fda7076de651820f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:32:06.622724+08:00, lastProcessTime=2025-04-08T21:31:56.624885+08:00 2025-04-08 21:32:06.680 INFO 26087 --- [ocp-schedules-4,45181cd2dfadf32f,e645332b513721e8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:32:06.737 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:32:06.622724+08:00 2025-04-08 21:32:08.449 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:08.450 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:08.493 INFO 26087 --- [ocp-schedules-5,5e6d2e7b36bef144,66a08bab07ddbbd7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:32:08.494 INFO 26087 --- [ocp-schedules-5,5e6d2e7b36bef144,66a08bab07ddbbd7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:32:08.587 INFO 26087 --- [ocp-async-1,e81e76887cd11ee4,bfd312720531cbd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:08.587 WARN 26087 --- [ocp-async-2,2406b63d9adad630,933116431f893bdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:09.271 INFO 26087 --- [prometheus-minute-schedule-1,21cb8fbd35254d5d,0497d156ae56f7bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:10.001 INFO 26087 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:32:10.001 INFO 26087 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:32:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:32:10.591 INFO 26087 --- [ocp-async-6,1cc9ed1cebbdec76,8388a99bfaeaa29d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:10.591 WARN 26087 --- [ocp-async-8,823fd9bfc54d8080,b8f97f718f6bc7e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:11.245 INFO 26087 --- [ocp-schedules-15,62db13194053f25d,3c58ad68908f285e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:32:11.450 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:11.450 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:12.271 INFO 26087 --- [prometheus-minute-schedule-1,78990df54765aa56,d03e15bc6362610e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:12.401 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119060, nodeCount=23, affectRows=23 2025-04-08 21:32:12.401 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119060, nodeCount=23, validCount=23 2025-04-08 21:32:12.404 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119060, nodeCount=24, affectRows=24 2025-04-08 21:32:12.404 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119060, nodeCount=24, validCount=24 2025-04-08 21:32:12.406 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119060, nodeCount=331, affectRows=331 2025-04-08 21:32:12.408 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119060, nodeCount=330, validCount=330 2025-04-08 21:32:12.591 WARN 26087 --- [ocp-async-7,863ff92b6ee3a5ed,03afddc9b8552a45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:12.591 INFO 26087 --- [ocp-async-6,0e20e7d2b29f03b7,08ff557856c1f64c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:14.451 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:14.451 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:14.592 WARN 26087 --- [ocp-async-2,d42e03018ac93c52,bd6580ebec5b0bd8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:14.592 INFO 26087 --- [ocp-async-5,8cc0cf176341ba3c,06cdbea984053013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:15.271 INFO 26087 --- [prometheus-minute-schedule-1,2167f4e81909819a,db9c1768e92a1a29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:16.433 INFO 26087 --- [ocp-schedules-12,805187427c9788cc,ca1aed01a4eaa439] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:32:16.434 INFO 26087 --- [ocp-schedules-10,b2f9b40625d15dfc,e1b9144b8418057f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:32:16.437 INFO 26087 --- [ocp-schedules-23,1fb10c18659a3961,479dd4d6bcf7ecbc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:32:16.437 INFO 26087 --- [ocp-schedules-23,1fb10c18659a3961,479dd4d6bcf7ecbc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:32:16.437 INFO 26087 --- [ocp-schedules-16,d79c209bc737c45a,c278c3d23bcdbd79] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:32:16.440 INFO 26087 --- [ocp-schedules-12,805187427c9788cc,ca1aed01a4eaa439] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:32:16.440 INFO 26087 --- [ocp-schedules-6,dce6b5e83d3a0f02,95b05097a52ca4bd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:32:16.440 INFO 26087 --- [ocp-schedules-6,dce6b5e83d3a0f02,95b05097a52ca4bd] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:32:16.440 INFO 26087 --- [ocp-schedules-23,e70f81e5e5d082e0,18b981421143a762] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:32:16.440 INFO 26087 --- [ocp-schedules-23,e70f81e5e5d082e0,18b981421143a762] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:32:16.440 INFO 26087 --- [ocp-schedules-10,b2f9b40625d15dfc,e1b9144b8418057f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:32:16.441 INFO 26087 --- [ocp-schedules-11,735eab6e6e0d388f,703b5bb38b788a6e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:32:16.441 INFO 26087 --- [ocp-schedules-3,70f87969d3a2da56,bad19fd3346006f4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:32:16.441 INFO 26087 --- [ocp-schedules-28,58d5064ea1593923,906737099d739762] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:32:16.466 INFO 26087 --- [ocp-schedules-15,77b25501e05f4d04,1f34a09f3a74ec4d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:32:16.481 INFO 26087 --- [ocp-async-3,c65520dd5638406f,3a912dc74901140c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:32:16.525 INFO 26087 --- [ocp-schedules-1,1fcc1fbe473c1d73,e74d517ef69840f6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:32:16.546 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:32:16.546 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:32:16.550 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:32:16.550 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:32:16.550 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:32:16.551 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:32:16.551 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:32:16.551 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:32:16.553 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:32:16.553 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:32:16.553 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:32:16.556 INFO 26087 --- [ocp-schedules-5,2cd3e4490cf328dd,252f71d67a7c0451] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:32:16.558 INFO 26087 --- [ocp-schedules-4,d7de7713eacc0af3,ff5f0fc8dc6a80c7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:32:16.566 INFO 26087 --- [ocp-schedules-13,6c6e33829d469309,07629eacddb3405b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:32:16.566 INFO 26087 --- [ocp-schedules-13,6c6e33829d469309,07629eacddb3405b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:32:16.575 INFO 26087 --- [ocp-schedules-24,0d6a688cfe3ee4c9,aed43c640fe34009] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:32:16.575+08:00, clusters=[] 2025-04-08 21:32:16.577 INFO 26087 --- [ocp-schedules-19,a16deddb3ccc0c74,2da2b60d77e8f8a5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:32:16.577 INFO 26087 --- [ocp-schedules-30,7bbc5e90391c1c0b,056d5e9c7a07a30a] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:32:16.577 INFO 26087 --- [ocp-schedules-30,7bbc5e90391c1c0b,056d5e9c7a07a30a] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:32:16.579 INFO 26087 --- [ocp-schedules-16,d62f7dc7bc074838,4353e0543301ed0c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:32:16.579+08:00, clusters=[] 2025-04-08 21:32:16.592 WARN 26087 --- [ocp-async-4,94bacdcc4a6a86a5,b5c15d28ebdfee98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:16.592 INFO 26087 --- [ocp-async-9,d957a2f931d56427,44c3b5e7bd670db5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:16.601 INFO 26087 --- [ocp-schedules-3,9ab0f62cbf4e1a99,2b064ce0323af5aa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:32:16.601 INFO 26087 --- [ocp-schedules-3,9ab0f62cbf4e1a99,2b064ce0323af5aa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:32:16.602 INFO 26087 --- [ocp-schedules-3,9ab0f62cbf4e1a99,2b064ce0323af5aa] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:32:16.602 INFO 26087 --- [ocp-schedules-3,9ab0f62cbf4e1a99,2b064ce0323af5aa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:32:16.605 INFO 26087 --- [ocp-schedules-11,bdc15e9558852241,ee0f3ff0cd901758] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:32:16.606 INFO 26087 --- [ocp-schedules-11,bdc15e9558852241,ee0f3ff0cd901758] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:32:16.606 INFO 26087 --- [ocp-schedules-11,bdc15e9558852241,ee0f3ff0cd901758] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:32:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:32:16.624029+08:00, lastProcessTime=2025-04-08T21:32:06.622724+08:00 2025-04-08 21:32:16.681 INFO 26087 --- [ocp-schedules-15,d081a9cf37191c56,50f6012f8c22de54] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:32:16.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:32:16.624029+08:00 2025-04-08 21:32:16.753 INFO 26087 --- [ocp-schedules-18,f802f6ad715fa052,166e03aba54d2080] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:32:16.767 INFO 26087 --- [ocp-schedules-29,2ce9f5e9cea62775,c22d7e5e1873bcfe] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:32:16.803 INFO 26087 --- [ocp-schedules-21,722376cb28cffae4,aaa09495bc938f35] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:32:16.803 INFO 26087 --- [ocp-schedules-21,722376cb28cffae4,aaa09495bc938f35] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:32:16.804 INFO 26087 --- [ocp-schedules-21,722376cb28cffae4,aaa09495bc938f35] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:32:16.804 INFO 26087 --- [ocp-schedules-21,722376cb28cffae4,aaa09495bc938f35] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:32:16.804 INFO 26087 --- [ocp-schedules-21,722376cb28cffae4,aaa09495bc938f35] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:32:16.804 INFO 26087 --- [ocp-schedules-21,722376cb28cffae4,aaa09495bc938f35] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:32:16.816 INFO 26087 --- [ocp-schedules-8,419872a778bbf884,9c49b20c53cd5d84] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:32:17.451 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:17.452 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:17.504 INFO 26087 --- [ocp-schedules-19,4f5ba47f48299319,ffa6e7df2b837a9a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:32:17.504 INFO 26087 --- [ocp-schedules-19,4f5ba47f48299319,ffa6e7df2b837a9a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:32:17.589 INFO 26087 --- [ocp-schedules-6,1344f91239add22d,4beefb2232c3e211] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:32:17.593 INFO 26087 --- [ocp-schedules-6,1344f91239add22d,4beefb2232c3e211] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:32:17.593 INFO 26087 --- [ocp-schedules-6,1344f91239add22d,4beefb2232c3e211] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:32:18.271 INFO 26087 --- [prometheus-minute-schedule-1,300787140248b04c,a13b0de693315ed0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:18.506 INFO 26087 --- [ocp-schedules-25,b4a209492801e232,63ddff19a69d1985] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:32:18.507 INFO 26087 --- [ocp-schedules-25,b4a209492801e232,63ddff19a69d1985] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:32:18.592 WARN 26087 --- [ocp-async-6,060bdd4712e2da55,e7b4d28bbba5088d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:18.592 INFO 26087 --- [ocp-async-10,dd3acf2323e410fe,9464a542b5ff0f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:20.452 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:20.453 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:32:20.592 WARN 26087 --- [ocp-async-1,a1b9bfb32c0db3f0,fdc71a600b76d09d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:20.592 INFO 26087 --- [ocp-async-3,795cdfc23147ad49,49114af8f151472e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:21.267 INFO 26087 --- [ocp-schedules-25,fecb1e9c3d260f32,d5b714ab369c07d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:32:21.271 INFO 26087 --- [prometheus-minute-schedule-1,eeadc1a6ade2aa56,06b9fc6dc700b493] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:22.592 WARN 26087 --- [ocp-async-4,5f4fd8503effbbf5,ed6b2eedcc8c16ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:22.592 INFO 26087 --- [ocp-async-8,44890f2f5bee208b,902ec74721dbd703] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:23.453 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:23.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:24.271 INFO 26087 --- [prometheus-minute-schedule-1,c560e8ae2b66c326,bd445198ff2d0ebb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:24.592 WARN 26087 --- [ocp-async-2,58ce91f58af37751,bb1480026f622e20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:24.592 INFO 26087 --- [ocp-async-9,79005b50c886adf7,68407d537e72c3ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:26.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:26.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:26.482 INFO 26087 --- [ocp-async-10,33a73740028e707a,310f0c34320016db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:32:26.592 WARN 26087 --- [ocp-async-5,a8ec0f10aca2113a,dc2c1c303659b2f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:26.592 INFO 26087 --- [ocp-async-3,c43f4ec619c97a41,889f1e2385092608] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:32:26.623517+08:00, lastProcessTime=2025-04-08T21:32:16.624029+08:00 2025-04-08 21:32:26.682 INFO 26087 --- [ocp-schedules-12,bbee997770255d30,fde4861e55ce5da0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:32:26.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:32:26.623517+08:00 2025-04-08 21:32:27.271 INFO 26087 --- [prometheus-minute-schedule-1,841c66ce5cac30dd,66875a3628f54a2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:28.518 INFO 26087 --- [ocp-schedules-30,c2e35744ad64abf5,36b14f7b6344c827] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:32:28.519 INFO 26087 --- [ocp-schedules-30,c2e35744ad64abf5,36b14f7b6344c827] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:32:28.593 WARN 26087 --- [ocp-async-2,e35b8e82e40237a3,10c2c47c3e652396] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:28.593 INFO 26087 --- [ocp-async-7,639ad120462b3086,cbe9742b734117e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:29.454 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:29.455 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:30.271 INFO 26087 --- [prometheus-minute-schedule-1,6064884d33c4f567,a7a95d23d7bacd54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:30.593 WARN 26087 --- [ocp-async-6,5e68645d9a8c1076,7bf9f73cb06cefda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:30.593 INFO 26087 --- [ocp-async-9,36d4e11cc8d53e9c,2d1ed5466d170566] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:31.288 INFO 26087 --- [ocp-schedules-19,c38e39756d257ac6,3f274c0c15de3b3e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:32:32.455 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:32.455 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:32.593 WARN 26087 --- [ocp-async-4,92def66d27f17e62,9c72be49c0f9b284] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:32.593 INFO 26087 --- [ocp-async-5,bdcfd5e0eca6f00c,723502f0e060fa3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:33.271 INFO 26087 --- [prometheus-minute-schedule-1,9a388ac490beb985,e94776d244fd330f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:34.593 WARN 26087 --- [ocp-async-2,7de33f6f37cedd3d,b3bcdb31530ca440] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:34.593 INFO 26087 --- [ocp-async-3,e5efcb8c38b63ec7,fccb928c1cafe8f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:35.455 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:35.456 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:36.271 INFO 26087 --- [prometheus-minute-schedule-1,1d3690c901425a13,fc5dfd7b04829584] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:36.482 INFO 26087 --- [ocp-async-7,291adce8915ffe0a,d62508dbc0e266d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:32:36.593 WARN 26087 --- [ocp-async-6,641c3f0dbb7510f7,64f80626d49b058d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:36.593 INFO 26087 --- [ocp-async-10,635196155989a0b4,82b83188235922b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:32:36.624053+08:00, lastProcessTime=2025-04-08T21:32:26.623517+08:00 2025-04-08 21:32:36.683 INFO 26087 --- [ocp-schedules-2,867f91f507df6ca0,20190533ce25ecad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:32:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:32:36.624053+08:00 2025-04-08 21:32:37.595 INFO 26087 --- [ocp-schedules-27,467eb84227ebc336,78d288eb05847deb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:32:37.599 INFO 26087 --- [ocp-schedules-27,467eb84227ebc336,78d288eb05847deb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:32:37.599 INFO 26087 --- [ocp-schedules-27,467eb84227ebc336,78d288eb05847deb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:32:38.456 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:38.457 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:38.526 INFO 26087 --- [ocp-schedules-8,d37a683e5e8e10fd,0d759d9066b12376] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:32:38.528 INFO 26087 --- [ocp-schedules-8,d37a683e5e8e10fd,0d759d9066b12376] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:32:38.593 INFO 26087 --- [ocp-async-1,bdaf85f05d5a3479,5bcd493a7d10f61f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:38.593 WARN 26087 --- [ocp-async-4,c00c5373d53b7da7,cf8e947d141a3cc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:39.271 INFO 26087 --- [prometheus-minute-schedule-1,a8afbb25651df901,c2023bf4bd020879] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:40.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:32:40.001 INFO 26087 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:32:40.593 WARN 26087 --- [ocp-async-2,4a82c546c5ccafe4,d04797dd86998d4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:40.594 INFO 26087 --- [ocp-async-2,a511c302d7a94565,f3255137c966f674] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:41.307 INFO 26087 --- [ocp-schedules-15,fa7bb98b8d1ecdce,e75b9dc12519adc6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:32:41.457 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:41.457 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:42.271 INFO 26087 --- [prometheus-minute-schedule-1,a1d357259c6d72d8,7b0b7a2b4dbab7a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:42.594 WARN 26087 --- [ocp-async-6,7a87fcf7271b06c5,261aafb9b0e10cd5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:42.594 INFO 26087 --- [ocp-async-2,5a7ae01e70da6a7d,f3f33f63c50b8b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:44.458 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:44.458 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:44.594 WARN 26087 --- [ocp-async-5,a3ed8cdbc38fef5c,604cd8dede627690] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:44.594 INFO 26087 --- [ocp-async-2,1ba249b22ca8d272,e558f5970e430c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:45.271 INFO 26087 --- [prometheus-minute-schedule-1,9fe985ea7a5dfd2e,012e4bc0ed71f038] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:46.437 INFO 26087 --- [ocp-schedules-4,150f685493b56dd8,3c9c717e72506616] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:32:46.438 INFO 26087 --- [ocp-schedules-11,3582e3796125eb48,ced3175f26224c34] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:32:46.438 INFO 26087 --- [ocp-schedules-11,3582e3796125eb48,ced3175f26224c34] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:32:46.440 INFO 26087 --- [ocp-schedules-22,ec1e8cd63077d431,3620d5ebd8a09704] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:32:46.440 INFO 26087 --- [ocp-schedules-22,ec1e8cd63077d431,3620d5ebd8a09704] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:32:46.440 INFO 26087 --- [ocp-schedules-10,cebf2aa5d7b871c5,48b6c260a40f31b7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:32:46.440 INFO 26087 --- [ocp-schedules-10,cebf2aa5d7b871c5,48b6c260a40f31b7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:32:46.441 INFO 26087 --- [ocp-schedules-13,0dca9c867bb201d0,5206480a678379e5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:32:46.441 INFO 26087 --- [ocp-schedules-18,f82422192d6d4468,7408bdd6874e9339] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:32:46.441 INFO 26087 --- [ocp-schedules-28,04afa6e0518d6bf2,7c51571cfa382cf6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:32:46.441 INFO 26087 --- [ocp-schedules-25,d969d8b09c84a3c3,b1f93b352dd6f184] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:32:46.482 INFO 26087 --- [ocp-async-2,51d8def8efec207b,675b3a29cade38d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:32:46.559 INFO 26087 --- [ocp-schedules-21,86fe1f3ffdd94c4c,c548bc16bf4b0ade] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:32:46.594 WARN 26087 --- [ocp-async-3,3454a01c42673047,d0f5023e18c2910e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:46.594 INFO 26087 --- [ocp-async-8,f4988d4ae5aecdd4,9bfb319106946f37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:32:46.623205+08:00, lastProcessTime=2025-04-08T21:32:36.624053+08:00 2025-04-08 21:32:46.684 INFO 26087 --- [ocp-schedules-30,04f2cae9921e1e13,b356f279275c6f84] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:32:46.743 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:32:46.623205+08:00 2025-04-08 21:32:47.458 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:47.459 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:48.271 INFO 26087 --- [prometheus-minute-schedule-1,72b7310f66151678,ca347a2f313457d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:48.532 INFO 26087 --- [ocp-schedules-8,0a84e241f529a0ac,ca815109a5ef96b6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:32:48.533 INFO 26087 --- [ocp-schedules-8,0a84e241f529a0ac,ca815109a5ef96b6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:32:48.594 WARN 26087 --- [ocp-async-10,7b8f58e94d17b2f5,ab39db0473737c11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:48.594 INFO 26087 --- [ocp-async-6,ec57b7a07632f131,79ad25a99c1e6cd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:50.459 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:50.460 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:50.594 WARN 26087 --- [ocp-async-4,0f0ee13ec3e499d9,4672ab59d1e3d503] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:50.594 INFO 26087 --- [ocp-async-2,9f4fe95d3d97aef9,b9b2b6c59ebbf679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:51.271 INFO 26087 --- [prometheus-minute-schedule-1,315a2e3f78a756cd,2ad4a323d7246910] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:51.335 INFO 26087 --- [ocp-schedules-26,3d472a533652f4e5,d8d1f8127db53817] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:32:52.594 WARN 26087 --- [ocp-async-3,fbeec5c177bdf984,d2f7cb2a76197e34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:52.594 INFO 26087 --- [ocp-async-1,699c7670b70053f2,d2809c0aa5861ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:53.460 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:53.460 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:54.271 INFO 26087 --- [prometheus-minute-schedule-1,d76057212c438bcd,b968c6efde8ed5f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:54.594 WARN 26087 --- [ocp-async-1,04d7a565991e8a32,5194e496b725d282] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:54.594 INFO 26087 --- [ocp-async-10,a1a30cef5c886f81,94c402f39c794870] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:56.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:56.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:32:56.482 INFO 26087 --- [ocp-async-9,a60131f4cf087191,9eab6656d3512785] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:32:56.496 INFO 26087 --- [ocp-schedules-14,1b3dfd729f4258f2,9205129e14e6df98] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:32:56.595 WARN 26087 --- [ocp-async-6,2cc68a4dafd6fcde,0ce1d0e5e67c2f6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:56.595 INFO 26087 --- [ocp-async-4,c5791d879b2b315d,bcae2e48d88d162c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:32:56.622816+08:00, lastProcessTime=2025-04-08T21:32:46.623205+08:00 2025-04-08 21:32:56.686 INFO 26087 --- [ocp-schedules-29,f921e18501371b4a,bd82ffc1c2f00664] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:32:56.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:32:56.622816+08:00 2025-04-08 21:32:57.271 INFO 26087 --- [prometheus-minute-schedule-1,18d74060d61ffa3f,54720f7afc108fd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119180, interval=60 2025-04-08 21:32:57.600 INFO 26087 --- [ocp-schedules-19,372d4cda044cb372,accb237fdde855f4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:32:57.604 INFO 26087 --- [ocp-schedules-19,372d4cda044cb372,accb237fdde855f4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:32:57.604 INFO 26087 --- [ocp-schedules-19,372d4cda044cb372,accb237fdde855f4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:32:58.539 INFO 26087 --- [ocp-schedules-20,93c318cee101bac0,e5a3afa233b5f0bd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:32:58.540 INFO 26087 --- [ocp-schedules-20,93c318cee101bac0,e5a3afa233b5f0bd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:32:58.595 WARN 26087 --- [ocp-async-4,09a6439e76365ef7,d3ad26bb841a91b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:32:58.595 INFO 26087 --- [ocp-async-7,8f1b9347d21379c7,21e754cee26e84fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:32:59.461 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:32:59.462 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:00.271 INFO 26087 --- [prometheus-minute-schedule-1,1179d795440d6d87,646852ac040aff02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:00.595 WARN 26087 --- [ocp-async-8,9b60e4b8e950ddaf,7f0830a35cfab2b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:00.595 INFO 26087 --- [ocp-async-1,337a77c8b9933c8d,3e4774c72305cfd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:01.355 INFO 26087 --- [ocp-schedules-28,df8c45cd527f540a,4a84c9cb0feb6fcd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:33:02.462 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:02.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:02.595 WARN 26087 --- [ocp-async-6,df632b650cc8d1f5,6995dca93c32188e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:02.595 INFO 26087 --- [ocp-async-5,6eaf9a0ba3cca5da,a51ecd0d3ba68796] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:03.271 INFO 26087 --- [prometheus-minute-schedule-1,5d3ebcebfa09d8bb,d54987c4a5245b84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:04.595 WARN 26087 --- [ocp-async-2,fa2652a964228fba,bb2b154e9831df37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:04.595 INFO 26087 --- [ocp-async-3,98ced0150aa451da,78c15976ab8bffef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:05.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:05.463 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:06.271 INFO 26087 --- [prometheus-minute-schedule-1,aff58f7357989902,28d65ed1b737bfd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:06.272 INFO 26087 --- [prometheus-cache-monitor1,ab0c94cef475f6b8,d1f3febe7c1e64bd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:33:06.272 INFO 26087 --- [prometheus-cache-monitor1,ab0c94cef475f6b8,d1f3febe7c1e64bd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:33:06.482 INFO 26087 --- [ocp-async-7,b0d6d57a461db98a,b9dd8bb11eec7284] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:33:06.595 WARN 26087 --- [ocp-async-10,c840018a37dd1ca0,bf7159d5e1fbba78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:06.595 INFO 26087 --- [ocp-async-9,afc718253183c61e,f7f7ec434e12e67d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:33:06.623102+08:00, lastProcessTime=2025-04-08T21:32:56.622816+08:00 2025-04-08 21:33:06.687 INFO 26087 --- [ocp-schedules-6,ab79a65eeab9d8af,c7493408da98224e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:33:06.703 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:33:06.623102+08:00 2025-04-08 21:33:08.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:08.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:08.543 INFO 26087 --- [ocp-schedules-15,e910de4a19c9bd5d,d9ceb487a764ced4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:33:08.545 INFO 26087 --- [ocp-schedules-15,e910de4a19c9bd5d,d9ceb487a764ced4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:33:08.596 WARN 26087 --- [ocp-async-6,e90dd3229a2d45fa,d7e96d3a6288ca41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:08.596 INFO 26087 --- [ocp-async-1,bbf67751e7ea5aa5,57763cce417bc689] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:09.271 INFO 26087 --- [prometheus-minute-schedule-1,7526407b9c25743c,8e6aa5ec2db58efb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:10.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:33:10.001 INFO 26087 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:33:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:33:10.596 INFO 26087 --- [ocp-async-1,3d796823b1832d4c,e1080e2e3c8515a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:10.596 WARN 26087 --- [ocp-async-4,2b737dea90e2cda4,4563d117694d76a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:11.375 INFO 26087 --- [ocp-schedules-19,28a812b207dbfb82,6049800caa8772c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:33:11.464 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:11.465 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:12.271 INFO 26087 --- [prometheus-minute-schedule-1,1b86945eda9edc9e,aa9b5c3763de48a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:12.415 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119120, nodeCount=73, affectRows=73 2025-04-08 21:33:12.416 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119120, nodeCount=73, validCount=73 2025-04-08 21:33:12.416 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119120, nodeCount=305, affectRows=305 2025-04-08 21:33:12.418 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119120, nodeCount=304, validCount=304 2025-04-08 21:33:12.596 WARN 26087 --- [ocp-async-8,a79c4eabfc08c760,6b9f559a52c8ca5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:12.596 INFO 26087 --- [ocp-async-7,72b5ea422de7c6ef,3af6505aa3faf4d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:14.465 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:14.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:14.596 WARN 26087 --- [ocp-async-6,4d36a1077603757e,a568381dba37897a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:14.596 INFO 26087 --- [ocp-async-5,813f6e8e67864ee6,a7c73b047d888375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:15.271 INFO 26087 --- [prometheus-minute-schedule-1,26bc3a0b8a5b579a,ccf452ca840e5a4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:16.433 INFO 26087 --- [ocp-schedules-6,a5d48b5f0b9e30aa,afa232287cb74b4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:33:16.434 INFO 26087 --- [ocp-schedules-15,d66b00ffc6bc5972,9f4db88c1dd4d66a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:33:16.437 INFO 26087 --- [ocp-schedules-10,1e9cbcbf91fcc26d,d1c168c8bc3e6bf7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:33:16.437 INFO 26087 --- [ocp-schedules-10,1e9cbcbf91fcc26d,d1c168c8bc3e6bf7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:33:16.437 INFO 26087 --- [ocp-schedules-11,97304768d5c8cbdd,d4f69d789a23e86a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:33:16.439 INFO 26087 --- [ocp-schedules-22,71ad5bc943cd9b50,40968fa582556c2f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:33:16.439 INFO 26087 --- [ocp-schedules-22,71ad5bc943cd9b50,40968fa582556c2f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:33:16.439 INFO 26087 --- [ocp-schedules-10,83aa6ecb2de794b6,d9300abdf87dd8d5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:33:16.439 INFO 26087 --- [ocp-schedules-10,83aa6ecb2de794b6,d9300abdf87dd8d5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:33:16.441 INFO 26087 --- [ocp-schedules-19,6b89082756938c42,ac3f3e49de9fd056] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:33:16.441 INFO 26087 --- [ocp-schedules-15,d66b00ffc6bc5972,9f4db88c1dd4d66a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:33:16.441 INFO 26087 --- [ocp-schedules-7,0de8adb2a8458d84,bee609a062600bab] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:33:16.441 INFO 26087 --- [ocp-schedules-26,01d4f6213973e178,3b5ee46508aab2d7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:33:16.442 INFO 26087 --- [ocp-schedules-6,a5d48b5f0b9e30aa,afa232287cb74b4c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:33:16.450 INFO 26087 --- [ocp-schedules-18,d55fdb4877c37de0,de4e9e6ba2ab239e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:33:16.452 INFO 26087 --- [ocp-schedules-9,6ebb190d1a739ea2,b4701ece96690286] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:33:16.453 INFO 26087 --- [ocp-schedules-13,08674e6eda26b0a6,085b58eedea15fc9] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:33:16.453 INFO 26087 --- [ocp-schedules-13,08674e6eda26b0a6,085b58eedea15fc9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:33:16.465 INFO 26087 --- [ocp-schedules-4,6f9c5f2170a9f351,dbb79ee63afd558c] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2025-04-08 21:33:16.466 INFO 26087 --- [ocp-schedules-4,6f9c5f2170a9f351,dbb79ee63afd558c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-04-08 21:33:16.470 INFO 26087 --- [pool-overview-container-7,dfe6a8bf6b911799,2c3ddcadd6fc377d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:33:16.470 INFO 26087 --- [pool-overview-container-6,411b3eb5589c1696,7c040d1d4c8c8249] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:33:16.482 INFO 26087 --- [ocp-async-4,d412baf8305469fd,670cd47c4bc0354f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:33:16.504 INFO 26087 --- [ocp-schedules-2,35213c5ba7acc9a9,f1429cb151c7c724] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:33:16.504 INFO 26087 --- [ocp-schedules-2,35213c5ba7acc9a9,f1429cb151c7c724] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:33:16.527 INFO 26087 --- [ocp-schedules-11,4c7e8bc75c0dc4e6,471a2d0ea3a63900] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:33:16.560 INFO 26087 --- [ocp-schedules-15,3f7885d89be1c066,318c8cf36e21c3bf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:33:16.562 INFO 26087 --- [ocp-schedules-22,d3cb665add42b663,d596b3d4e92526b7] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 21:33:16.562 INFO 26087 --- [ocp-schedules-22,d3cb665add42b663,d596b3d4e92526b7] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 21:33:16.562 INFO 26087 --- [ocp-schedules-22,d3cb665add42b663,d596b3d4e92526b7] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=428 2025-04-08 21:33:16.568 INFO 26087 --- [ocp-schedules-19,3df21eac20f36879,b7334149312e9124] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:33:16.568 INFO 26087 --- [ocp-schedules-19,3df21eac20f36879,b7334149312e9124] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:33:16.572 INFO 26087 --- [ocp-schedules-22,d3cb665add42b663,d596b3d4e92526b7] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=428 2025-04-08 21:33:16.577 INFO 26087 --- [ocp-schedules-7,c640fff2458a0f73,dced30742ff6adef] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:33:16.577+08:00, clusters=[] 2025-04-08 21:33:16.579 INFO 26087 --- [ocp-schedules-26,6cd3b48060f15fa2,b71c3c6ff116e9cd] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:33:16.579 INFO 26087 --- [ocp-schedules-26,27e7088fcda0af28,2292c2aee65072f4] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:33:16.579 INFO 26087 --- [ocp-schedules-26,27e7088fcda0af28,2292c2aee65072f4] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:33:16.580 INFO 26087 --- [ocp-schedules-8,dd680cd481796638,2eebce3557460767] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:33:16.579+08:00, clusters=[] 2025-04-08 21:33:16.596 WARN 26087 --- [ocp-async-1,52797acc4d8f2894,5f4a8c686d507bb6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:16.596 INFO 26087 --- [ocp-async-9,4e86ef808de88b8c,c9701747ba2d370a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:16.603 INFO 26087 --- [ocp-schedules-9,d6677100f6a64d27,146cdb1e01e68c1f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:33:16.603 INFO 26087 --- [ocp-schedules-9,d6677100f6a64d27,146cdb1e01e68c1f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:33:16.604 INFO 26087 --- [ocp-schedules-9,d6677100f6a64d27,146cdb1e01e68c1f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:33:16.604 INFO 26087 --- [ocp-schedules-9,d6677100f6a64d27,146cdb1e01e68c1f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:33:16.606 INFO 26087 --- [ocp-schedules-13,84174d711196230b,a33402a9c32b568b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:33:16.607 INFO 26087 --- [ocp-schedules-13,84174d711196230b,a33402a9c32b568b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:33:16.607 INFO 26087 --- [ocp-schedules-13,84174d711196230b,a33402a9c32b568b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 0 2025-04-08 21:33:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:33:16.622883+08:00, lastProcessTime=2025-04-08T21:33:06.623102+08:00 2025-04-08 21:33:16.688 INFO 26087 --- [ocp-schedules-3,60081b160171a0f3,6379461b6b2f9bbc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:33:16.689 INFO 26087 --- [ocp-schedules-16,be5c4c4c2b6cb29e,a9d527bf1235869f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 21:33:16.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:33:16.622883+08:00 2025-04-08 21:33:16.743 INFO 26087 --- [ocp-schedules-30,cab779ec354e35b6,4a65c2b8128d788b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:33:16.743 INFO 26087 --- [ocp-schedules-30,cab779ec354e35b6,4a65c2b8128d788b] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:33:16.745 INFO 26087 --- [ocp-schedules-30,cab779ec354e35b6,4a65c2b8128d788b] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:33:16.753 INFO 26087 --- [ocp-schedules-20,a155ff788f7b01f2,cfd956286e0b82a6] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:33:16.754 INFO 26087 --- [ocp-schedules-14,5ff20f46c5306408,6221d518b40dddc6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:33:16.768 INFO 26087 --- [ocp-schedules-21,e2b034aca3f7503e,bce1931c8c8bbc51] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:33:16.804 INFO 26087 --- [ocp-schedules-25,c800fa8488098148,b88a0f1eb66d613f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:33:16.804 INFO 26087 --- [ocp-schedules-25,c800fa8488098148,b88a0f1eb66d613f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:33:16.805 INFO 26087 --- [ocp-schedules-25,c800fa8488098148,b88a0f1eb66d613f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:33:16.805 INFO 26087 --- [ocp-schedules-25,c800fa8488098148,b88a0f1eb66d613f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:33:16.805 INFO 26087 --- [ocp-schedules-25,c800fa8488098148,b88a0f1eb66d613f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:33:16.805 INFO 26087 --- [ocp-schedules-25,c800fa8488098148,b88a0f1eb66d613f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:33:16.831 INFO 26087 --- [ocp-schedules-27,269b47f3bf98de9c,550ed1366244a278] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:33:17.078 INFO 26087 --- [ocp-schedules-24,469495def176dc2a,892f48e65c0d96bc] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:33:17.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:17.466 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:17.507 INFO 26087 --- [ocp-schedules-15,ad01d734ce2bae9e,8e825e6f2fc36031] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[] 2025-04-08 21:33:17.507 INFO 26087 --- [ocp-schedules-15,ad01d734ce2bae9e,8e825e6f2fc36031] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-04-08 21:33:17.606 INFO 26087 --- [ocp-schedules-1,b9f4e9b53027290a,1e185e1846784603] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:33:17.654 INFO 26087 --- [ocp-schedules-1,b9f4e9b53027290a,1e185e1846784603] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:33:17.654 INFO 26087 --- [ocp-schedules-1,b9f4e9b53027290a,1e185e1846784603] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:33:18.271 INFO 26087 --- [prometheus-minute-schedule-1,7c72cd602b4daa4d,fdd391bdee5fe107] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:18.548 INFO 26087 --- [ocp-schedules-11,74862303f5c1f7a1,7bd899108224ab8e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:33:18.549 INFO 26087 --- [ocp-schedules-11,74862303f5c1f7a1,7bd899108224ab8e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:33:18.597 WARN 26087 --- [ocp-async-5,d07edbeb3298c9eb,e24257e6e988194b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:18.597 INFO 26087 --- [ocp-async-6,f20a91c3f9b8b2ff,645564ec0dd17d09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:20.467 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:20.467 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:33:20.597 WARN 26087 --- [ocp-async-4,0e8e6fdd6a2da790,c356c601f9d15689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:20.597 INFO 26087 --- [ocp-async-6,94837ad9e5a2d7b6,27deb06bd1f4c688] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:21.271 INFO 26087 --- [prometheus-minute-schedule-1,76a0a17512197b4c,cc4de949e8d02ab9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:21.400 INFO 26087 --- [ocp-schedules-17,530d77ed0411a071,1a876ae7c4c183d3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:33:22.597 WARN 26087 --- [ocp-async-1,53cff43ce03e7e77,2cde1b87fddc0880] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:22.597 INFO 26087 --- [ocp-async-8,aa4f16012f50f1ad,acd23525c2cdc233] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:22.915 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0385806035688a68,786b11d468706a05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.11.12.102, traceId=0385806035688a68, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,OBCLUSTER,] 2025-04-08 21:33:22.921 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,0385806035688a68,786b11d468706a05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=0385806035688a68, duration=10 ms] 2025-04-08 21:33:22.949 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,40b9b03fa81602bf,372a217f17b37ded] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.11.12.102, traceId=40b9b03fa81602bf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:33:22.953 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,40b9b03fa81602bf,372a217f17b37ded] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.11.12.102, traceId=40b9b03fa81602bf, duration=22 ms] 2025-04-08 21:33:23.371 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,389f472e1c9d46fa,b570c737be127d3d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A33%3A22%2B08%3A00&endTime=2025-04-08T21%3A33%3A22%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=389f472e1c9d46fa, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:33:22+08:00,2025-04-08T21:33:22+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:33:23.375 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,389f472e1c9d46fa,b570c737be127d3d] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = tps 2025-04-08 21:33:23.375 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-9,389f472e1c9d46fa,b570c737be127d3d] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = tps, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744115610 2025-04-08 21:33:23.376 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,389f472e1c9d46fa,b570c737be127d3d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=389f472e1c9d46fa, duration=10 ms] 2025-04-08 21:33:23.387 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,0ad2e9b919f3f189,4faf772495395dbd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A33%3A22%2B08%3A00&endTime=2025-04-08T21%3A33%3A22%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=0ad2e9b919f3f189, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:33:22+08:00,2025-04-08T21:33:22+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:33:23.387 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,0ad2e9b919f3f189,4faf772495395dbd] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:33:23.387 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,0ad2e9b919f3f189,4faf772495395dbd] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744115610 2025-04-08 21:33:23.388 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,0ad2e9b919f3f189,4faf772495395dbd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=0ad2e9b919f3f189, duration=4 ms] 2025-04-08 21:33:23.399 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,71de99a5934ac401,f28ad8a3ebd9bf63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A33%3A22%2B08%3A00&endTime=2025-04-08T21%3A33%3A22%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=71de99a5934ac401, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:33:22+08:00,2025-04-08T21:33:22+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:33:23.400 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,71de99a5934ac401,f28ad8a3ebd9bf63] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_rt 2025-04-08 21:33:23.400 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-10,71de99a5934ac401,f28ad8a3ebd9bf63] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_rt, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744115610 2025-04-08 21:33:23.402 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,71de99a5934ac401,f28ad8a3ebd9bf63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=71de99a5934ac401, duration=6 ms] 2025-04-08 21:33:23.432 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,6f3aec6a51ff1f72,982997397d65c783] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-04-08T20%3A33%3A22%2B08%3A00&endTime=2025-04-08T21%3A33%3A22%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.11.12.102, traceId=6f3aec6a51ff1f72, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:33:22+08:00,2025-04-08T21:33:22+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-04-08 21:33:23.433 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-8,6f3aec6a51ff1f72,982997397d65c783] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = tps_rt 2025-04-08 21:33:23.433 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-8,6f3aec6a51ff1f72,982997397d65c783] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = tps_rt, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744115610 2025-04-08 21:33:23.433 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,6f3aec6a51ff1f72,982997397d65c783] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=6f3aec6a51ff1f72, duration=5 ms] 2025-04-08 21:33:23.467 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:23.468 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:23.899 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,c044c89c8630e344,1c683d3587c870b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=c044c89c8630e344, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:33:23.901 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,c044c89c8630e344,1c683d3587c870b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=c044c89c8630e344, duration=6 ms] 2025-04-08 21:33:24.271 INFO 26087 --- [prometheus-minute-schedule-1,c959447fdbe0df29,0ed5529381f2ff71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:24.314 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,8c8e08b44e49a25d,3433ac06cb7717ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.11.12.102, traceId=8c8e08b44e49a25d, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-04-08 21:33:24.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,15c7086d7f1bfea1,0205d5b4c934b526] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.11.12.102, traceId=15c7086d7f1bfea1, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-04-08 21:33:24.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,98e22d564ca501df,76789b8b103b798a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.11.12.102, traceId=98e22d564ca501df, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-04-08 21:33:24.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,62d88d05bde2c9e7,16c8380d829b4f69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.11.12.102, traceId=62d88d05bde2c9e7, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-04-08 21:33:24.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,15c7086d7f1bfea1,0205d5b4c934b526] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.11.12.102, traceId=15c7086d7f1bfea1, duration=3 ms] 2025-04-08 21:33:24.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,62d88d05bde2c9e7,16c8380d829b4f69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.11.12.102, traceId=62d88d05bde2c9e7, duration=2 ms] 2025-04-08 21:33:24.315 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,3b28b9c66e97a957,6cdc44b6fe2d8ed7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-04-08T20%3A33%3A24%2B08%3A00&endTime=2025-04-08T21%3A33%3A24%2B08%3A00&maxPoints=360, client=10.11.12.102, traceId=3b28b9c66e97a957, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-04-08T20:33:24+08:00,2025-04-08T21:33:24+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-04-08 21:33:24.316 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-1,3b28b9c66e97a957,6cdc44b6fe2d8ed7] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = sql_all_count 2025-04-08 21:33:24.316 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-1,3b28b9c66e97a957,6cdc44b6fe2d8ed7] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = sql_all_count, minTimestamp = 0, minTimeBuffer = 0, startTime = 1744115610 2025-04-08 21:33:24.316 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,8c8e08b44e49a25d,3433ac06cb7717ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.11.12.102, traceId=8c8e08b44e49a25d, duration=5 ms] 2025-04-08 21:33:24.317 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,3b28b9c66e97a957,6cdc44b6fe2d8ed7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.11.12.102, traceId=3b28b9c66e97a957, duration=6 ms] 2025-04-08 21:33:24.319 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,98e22d564ca501df,76789b8b103b798a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.11.12.102, traceId=98e22d564ca501df, duration=8 ms] 2025-04-08 21:33:24.353 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,8f8f34931fd4acba,bb708ac6847a7c43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.11.12.102, traceId=8f8f34931fd4acba, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-04-08 21:33:24.359 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,8f8f34931fd4acba,bb708ac6847a7c43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.11.12.102, traceId=8f8f34931fd4acba, duration=26 ms] 2025-04-08 21:33:24.597 WARN 26087 --- [ocp-async-8,e440c13c71594891,7e087fc45a63c5f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:24.597 INFO 26087 --- [ocp-async-5,b815b69fc83a085d,5b9b36eb013a4862] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:25.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,c45d7af405fbf68c,1d55775af456a70e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=c45d7af405fbf68c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:33:25.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,c45d7af405fbf68c,1d55775af456a70e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=c45d7af405fbf68c, duration=6 ms] 2025-04-08 21:33:26.468 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:26.469 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:26.482 INFO 26087 --- [ocp-async-7,9d793a63ad22e4ef,daf5df38afe8dcd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:33:26.597 WARN 26087 --- [ocp-async-6,f7a521db5486f17c,63c005b4fe155b89] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:26.597 INFO 26087 --- [ocp-async-2,d84a97c4e12caea2,4fcf9e3c7ff88a4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:33:26.622925+08:00, lastProcessTime=2025-04-08T21:33:16.622883+08:00 2025-04-08 21:33:26.689 INFO 26087 --- [ocp-schedules-26,1032e9034e5ea7ef,fe1f800b70e26994] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:33:26.749 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:33:26.622925+08:00 2025-04-08 21:33:27.271 INFO 26087 --- [prometheus-minute-schedule-1,4298c6231508f6db,9fbfabaa2f74facc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:28.554 INFO 26087 --- [ocp-schedules-24,f19d3803a9397524,547d78db103aa4be] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:33:28.555 INFO 26087 --- [ocp-schedules-24,f19d3803a9397524,547d78db103aa4be] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:33:28.597 INFO 26087 --- [ocp-async-2,9fe259320819c83e,cd0aaa3d8e4fe019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:28.597 WARN 26087 --- [ocp-async-3,e1e848c03b893536,45fb7ad2ec39d921] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:28.607 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:33:28.607 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:33:28.610 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:33:28.610 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:33:28.610 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:33:28.610 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:33:28.613 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:33:28.613 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:33:28.613 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:33:28.613 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:33:28.636 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:33:28.641 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:33:28.641 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:33:28.641 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:33:28.641 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:33:28.641 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:33:28.646 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:33:28.938 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:33:29.243 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.629552662S 2025-04-08 21:33:29.243 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:33:29.296 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:33:29.296 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:33:29.320 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:33:29.355 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:33:29.355 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:33:29.355 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:33:29.355 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:33:29.355 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:33:29.355 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:33:29.363 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:33:29.466 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] o.s.boot.SpringApplication : Started application in 0.856 seconds (JVM running for 1914.922) 2025-04-08 21:33:29.466 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.856318989S 2025-04-08 21:33:29.469 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:29.469 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:29.733 INFO 26087 --- [ocp-schedules-4,42032aaefba11d23,abb26f9305ee9b68] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:33:29.832 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,37bbf546d61ed414,66092bb07e804ede] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=37bbf546d61ed414, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:33:29.832 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,a4cc50b3393032bd,581059abcfb76836] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=a4cc50b3393032bd, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:33:29.833 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,d1fab7e278386a69,cf21fe79a370505f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=d1fab7e278386a69, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:33:29.833 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,d1fab7e278386a69,cf21fe79a370505f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=d1fab7e278386a69, duration=5 ms] 2025-04-08 21:33:29.833 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,9716079239f47752,4e794159f49de930] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=9716079239f47752, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:33:29.833 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,a4cc50b3393032bd,581059abcfb76836] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=a4cc50b3393032bd, duration=5 ms] 2025-04-08 21:33:29.835 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,37bbf546d61ed414,66092bb07e804ede] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=37bbf546d61ed414, duration=7 ms] 2025-04-08 21:33:29.930 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,9716079239f47752,4e794159f49de930] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=9716079239f47752, duration=101 ms] 2025-04-08 21:33:30.017 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:33:30.017 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:33:30.017 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:33:30.019 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,4f351f47576f0d75,be16cba71939861d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=4f351f47576f0d75, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:33:30.024 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,4f351f47576f0d75,be16cba71939861d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=4f351f47576f0d75, duration=8 ms] 2025-04-08 21:33:30.271 INFO 26087 --- [prometheus-minute-schedule-1,9a30f4071d5be14d,d680e8f2266f3713] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:30.597 WARN 26087 --- [ocp-async-10,224a4a298dfdcbe2,e4f0802529abffa6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:30.597 INFO 26087 --- [ocp-async-7,63e3a4893fef045a,e2d66092518bdbc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:31.426 INFO 26087 --- [ocp-schedules-18,667770ed6e732eba,cd951521e622bea8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:33:31.821 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,f023af89e3bd46e8,fdaaea26b5e749d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=f023af89e3bd46e8, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:33:31.821 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,f023af89e3bd46e8,fdaaea26b5e749d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=f023af89e3bd46e8, duration=2 ms] 2025-04-08 21:33:31.821 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c19a0fcd806c5e5,f15e63e7e3eac6e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=8c19a0fcd806c5e5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:33:31.821 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,8c19a0fcd806c5e5,f15e63e7e3eac6e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=8c19a0fcd806c5e5, duration=2 ms] 2025-04-08 21:33:31.822 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2b83cff12520189c,d4fd770c115ff037] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=2b83cff12520189c, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:33:31.822 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,2b83cff12520189c,d4fd770c115ff037] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=2b83cff12520189c, duration=3 ms] 2025-04-08 21:33:31.840 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:33:31.840 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:33:31.840 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:33:32.469 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:32.470 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:32.598 WARN 26087 --- [ocp-async-4,bf39d89234a04982,add0eecd7d664eed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:32.598 INFO 26087 --- [ocp-async-6,06060d08ca459b1a,78d5f4368102637c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:33.271 INFO 26087 --- [prometheus-minute-schedule-1,f40eb53baec2dc1c,63d00236b188ccbb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:34.598 WARN 26087 --- [ocp-async-8,d565c33535221473,20f3989bd83cdb05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:34.598 INFO 26087 --- [ocp-async-9,648b464ebda480fe,b460eb8e1160461b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:34.817 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,8a1d5b1a9fca208b,a5d5391df7069024] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=8a1d5b1a9fca208b, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:33:34.819 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,8a1d5b1a9fca208b,a5d5391df7069024] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=8a1d5b1a9fca208b, duration=5 ms] 2025-04-08 21:33:34.912 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:34.912 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:33:34.913 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:34.913 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:33:34.913 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:33:34.914 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1ms, period=120000ms 2025-04-08 21:33:35.470 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:35.471 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:36.271 INFO 26087 --- [prometheus-minute-schedule-1,2b23edf78b891d11,99da775a433695f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:36.483 INFO 26087 --- [ocp-async-3,6415bda9a2bca926,97eb831d67a64e61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:33:36.598 WARN 26087 --- [ocp-async-5,72cd9e2d323cbbab,e9e272c24702153c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:36.598 INFO 26087 --- [ocp-async-10,11909ddc5db4935c,e6425b48b69f6b0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:33:36.623107+08:00, lastProcessTime=2025-04-08T21:33:26.622925+08:00 2025-04-08 21:33:36.690 INFO 26087 --- [ocp-schedules-27,5130e991b5729548,0fc4704542d84d99] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:33:36.717 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:33:36.623107+08:00 2025-04-08 21:33:37.656 INFO 26087 --- [ocp-schedules-19,76c62ce77059588d,e02de291ad24bbb8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:33:37.660 INFO 26087 --- [ocp-schedules-19,76c62ce77059588d,e02de291ad24bbb8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:33:37.660 INFO 26087 --- [ocp-schedules-19,76c62ce77059588d,e02de291ad24bbb8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:33:38.471 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:38.471 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:38.560 INFO 26087 --- [ocp-schedules-8,3e99efe6eb01d2c5,52a3c3c82ac286ba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:33:38.561 INFO 26087 --- [ocp-schedules-8,3e99efe6eb01d2c5,52a3c3c82ac286ba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:33:38.598 WARN 26087 --- [ocp-async-6,b3591c5f672b2957,0f72d51d19538f71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:38.598 INFO 26087 --- [ocp-async-4,094014fd9fc2fa17,8ca8052086893843] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:39.271 INFO 26087 --- [prometheus-minute-schedule-1,6c9d20a3260af0da,b6449a153cf744d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:40.001 INFO 26087 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:33:40.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:33:40.598 WARN 26087 --- [ocp-async-9,809a11bb06c1cd03,b385876dc391ec12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:40.598 INFO 26087 --- [ocp-async-8,b15b8a22945acebd,ef81d5e86257a72d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:41.446 INFO 26087 --- [ocp-schedules-8,2049a3eb25684d96,926a06a7b9e30a9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:33:41.471 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:41.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:42.271 INFO 26087 --- [prometheus-minute-schedule-1,1334a0f5c562d2ce,ad235ac2ee56f390] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:42.598 WARN 26087 --- [ocp-async-7,9aed64507a09b3e3,d7549bfc4940d86a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:42.598 INFO 26087 --- [ocp-async-2,3e52012f2f45c7b0,d28f820410784700] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:44.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:44.472 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:44.598 INFO 26087 --- [ocp-async-10,60ab09f86c71b1b0,84c74f377168b628] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:44.599 WARN 26087 --- [ocp-async-1,d3f36c8411036a13,343fd074ac14c02d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:45.271 INFO 26087 --- [prometheus-minute-schedule-1,ddf8c2b3fbe2942e,2421bd8167dbc13a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:46.437 INFO 26087 --- [ocp-schedules-17,ed28b9086f6dc985,84aed3adb8d0d3bb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:33:46.437 INFO 26087 --- [ocp-schedules-17,ed28b9086f6dc985,84aed3adb8d0d3bb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:33:46.437 INFO 26087 --- [ocp-schedules-17,2766c81e787d2372,2ed1e71e46abe19c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:33:46.440 INFO 26087 --- [ocp-schedules-25,1197b44adcf5268f,78bd7be62f7af456] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:33:46.440 INFO 26087 --- [ocp-schedules-25,1197b44adcf5268f,78bd7be62f7af456] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:33:46.440 INFO 26087 --- [ocp-schedules-18,f9182a44c0e69b14,07194d0c2d5c1c33] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:33:46.440 INFO 26087 --- [ocp-schedules-18,f9182a44c0e69b14,07194d0c2d5c1c33] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:33:46.441 INFO 26087 --- [ocp-schedules-23,2ecab165b75bd27e,5cd04b91a5041836] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:33:46.441 INFO 26087 --- [ocp-schedules-3,dd556d0ab312509e,a1e126b1b3e17303] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:33:46.441 INFO 26087 --- [ocp-schedules-27,14f6ae4b2d14a33b,fff2110e9e9defca] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:33:46.483 INFO 26087 --- [ocp-async-6,cad646602c544a43,bfa54d49bda2a527] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:33:46.506 INFO 26087 --- [ocp-schedules-10,5db389b4d89d0568,ee98a4a5cae7de06] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:33:46.562 INFO 26087 --- [ocp-schedules-4,442e4303f55a53b4,edfdea330a0ccdec] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:33:46.599 WARN 26087 --- [ocp-async-3,71b45fc8e7381867,79fb351fbc8ac8b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:46.599 INFO 26087 --- [ocp-async-4,616906ebc496cf6e,25a4c2a6c2d5148a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:33:46.623911+08:00, lastProcessTime=2025-04-08T21:33:36.623107+08:00 2025-04-08 21:33:46.691 INFO 26087 --- [ocp-schedules-28,43cd10565bca9562,6bd88955fc725e86] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:33:46.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:33:46.623911+08:00 2025-04-08 21:33:47.473 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:47.473 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:48.271 INFO 26087 --- [prometheus-minute-schedule-1,46fb09ae12c47e66,cc8172dccfdbcc46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:48.566 INFO 26087 --- [ocp-schedules-7,832d794477ed9dbf,a8a96bdde1a3f228] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:33:48.567 INFO 26087 --- [ocp-schedules-7,832d794477ed9dbf,a8a96bdde1a3f228] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:33:48.599 WARN 26087 --- [ocp-async-4,2ed5ce227458b10b,b547d1574a20b9eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:48.599 INFO 26087 --- [ocp-async-7,0f0cc4c5ff3b7dc2,0d2c1c7b93801b0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:48.860 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOverPreCheck, client 10.11.12.102, traceId 423c563aa26f07ef] 2025-04-08 21:33:48.862 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOverPreCheck?null, client=10.11.12.102, traceId=423c563aa26f07ef, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverClusterPreCheck(TakeOverClusterPreCheckParam), args=TakeOverClusterPreCheckParam(connectionMode=direct, clusterName=null, vpcName=null, vpcId=1, obClusterId=null, rootSyspassword=xxx proxyropassword=xxx saveToCredential=true, address=192.168.1.61, port=2881),] 2025-04-08 21:33:48.869 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.869 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.870 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.871 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.871 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.872 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.872 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.873 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.873 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.873 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.877 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.877 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 21:33:48.883 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.883 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.884 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.884 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 21:33:48.890 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.890 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.891 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.891 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 21:33:48.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.897 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.898 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.899 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.902 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.903 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 21:33:48.903 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.903 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.904 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.904 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.905 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.906 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 21:33:48.912 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.912 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.916 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.916 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 21:33:48.917 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.918 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.921 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.923 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 21:33:48.924 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.925 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.925 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.926 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.929 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:33:48.929 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.932 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:33:48.936 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.937 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.940 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.940 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:33:48.955 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.955 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.956 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.957 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.958 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.959 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.960 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.960 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:33:48.975 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.975 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.979 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.979 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:33:48.980 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.980 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.981 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.982 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:33:48.996 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.997 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.997 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.998 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:48.998 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:48.999 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.000 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.000 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.001 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.001 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:33:49.016 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.017 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.017 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.018 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:33:49.018 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.019 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.020 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.020 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:33:49.037 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.037 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.038 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.039 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.039 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.040 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.040 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.041 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.042 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.042 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:33:49.056 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.056 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.057 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.058 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:33:49.061 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.061 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.062 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.062 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.063 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.063 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.064 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.064 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 21:33:49.074 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.074 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.076 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.077 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.077 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:33:49.078 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.078 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:33:49.078 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.079 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:33:49.084 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.084 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.085 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.085 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.086 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.086 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.087 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.087 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:33:49.088 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:33:49.088 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:33:49.097 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 21:33:49.098 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,423c563aa26f07ef,bde7088b81b123e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOverPreCheck client=10.11.12.102, traceId=423c563aa26f07ef, duration=240 ms] 2025-04-08 21:33:49.278 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,c992d337537fad41,943254524e8f8f5b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.11.12.102, traceId=c992d337537fad41, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-04-08 21:33:49.278 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,7391629481b451d2,4c6f88c83c6c48ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me/credentials?targetType=HOST, client=10.11.12.102, traceId=7391629481b451d2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.CredentialController.listCredentials(AccessTargetType,String,Pageable), args=HOST,Page request [number: 0, size 5000, sort: createTime: DESC],] 2025-04-08 21:33:49.280 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,c992d337537fad41,943254524e8f8f5b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.11.12.102, traceId=c992d337537fad41, duration=6 ms] 2025-04-08 21:33:49.282 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,7391629481b451d2,4c6f88c83c6c48ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me/credentials client=10.11.12.102, traceId=7391629481b451d2, duration=8 ms] 2025-04-08 21:33:50.473 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:50.474 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:50.599 WARN 26087 --- [ocp-async-7,48ce7b138955d528,998c76bb6d6055b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:50.599 INFO 26087 --- [ocp-async-10,9977c48eb06a27d3,5e3d0accf09e0f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:51.271 INFO 26087 --- [prometheus-minute-schedule-1,9a22aaad8e82322e,4d182772244a12c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:51.471 INFO 26087 --- [ocp-schedules-30,b7e9281e4992a220,aa8af199ec0c0e1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:33:52.599 WARN 26087 --- [ocp-async-3,c4fde49934824579,b0b277eaa2a13290] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:52.599 INFO 26087 --- [ocp-async-8,1356d58e476d3561,2834ffeb640d3d0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:53.474 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:53.475 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:54.271 INFO 26087 --- [prometheus-minute-schedule-1,46a57bc4c7d3cd6e,6f409e926093e052] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:54.599 WARN 26087 --- [ocp-async-4,f8c1a10db64671cd,ce85c44d2c2b1d38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:54.599 INFO 26087 --- [ocp-async-2,7988a3dfecf2976d,ced6e6af308094e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:56.041 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,0d7d4f1ef0a38143,9f93809519763205] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId 0d7d4f1ef0a38143] 2025-04-08 21:33:56.043 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,0d7d4f1ef0a38143,9f93809519763205] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=0d7d4f1ef0a38143, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 21:33:56.047 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,0d7d4f1ef0a38143,9f93809519763205] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)] 2025-04-08 21:33:56.048 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:33:56.063 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.104 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:33:56.119 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.119 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:33:56.134 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.134 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.134 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:33:56.149 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.149 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.61 2025-04-08 21:33:56.174 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.174 INFO 26087 --- [pool-27-thread-1,0d7d4f1ef0a38143,ef25d97ab0a14c41] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:33:56.174 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,0d7d4f1ef0a38143,9f93809519763205] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=0d7d4f1ef0a38143, duration=135 ms] 2025-04-08 21:33:56.475 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:56.475 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:33:56.483 INFO 26087 --- [ocp-async-5,a32aa010a8cd1833,a26229869f634065] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:33:56.497 INFO 26087 --- [ocp-schedules-28,21b79dc07e5cd257,ee3e0476c1378fad] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:33:56.599 WARN 26087 --- [ocp-async-1,91327fbde6a2f072,94f3251a9c834139] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:56.599 INFO 26087 --- [ocp-async-7,4a85fb128cf8bd31,b2696ef7cf81958c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:33:56.623445+08:00, lastProcessTime=2025-04-08T21:33:46.623911+08:00 2025-04-08 21:33:56.693 INFO 26087 --- [ocp-schedules-13,1ff29b152f2fd98a,d2440cd3858cc74a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:33:56.726 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:33:56.623445+08:00 2025-04-08 21:33:57.271 INFO 26087 --- [prometheus-minute-schedule-1,4ebbf907bbb22493,111eb6775d8b4f68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119240, interval=60 2025-04-08 21:33:57.661 INFO 26087 --- [ocp-schedules-10,027fd8be4482d90f,3d0a8e30df84a57b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:33:57.667 INFO 26087 --- [ocp-schedules-10,027fd8be4482d90f,3d0a8e30df84a57b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:33:57.667 INFO 26087 --- [ocp-schedules-10,027fd8be4482d90f,3d0a8e30df84a57b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:33:58.572 INFO 26087 --- [ocp-schedules-18,dafed894d0abf5af,5ba32afde6279fdc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:33:58.573 INFO 26087 --- [ocp-schedules-18,dafed894d0abf5af,5ba32afde6279fdc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:33:58.600 WARN 26087 --- [ocp-async-9,5d346951f2e19859,aa62c5c4440e0849] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:33:58.600 INFO 26087 --- [ocp-async-10,d2dcbf6c7280bdb1,dd55e98b3ed1401e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:33:59.475 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:33:59.476 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:00.271 INFO 26087 --- [prometheus-minute-schedule-1,04f58085c246f30b,4ef65ecb9ea4d722] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:00.600 WARN 26087 --- [ocp-async-10,bd87634733afc5b3,a106ee7d175868f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:00.600 INFO 26087 --- [ocp-async-5,c9c002b1b407c74d,161ebdc592353cc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:01.490 INFO 26087 --- [ocp-schedules-1,931b416b44197fa3,630ca090c64dd8aa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:34:02.476 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:02.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:02.600 WARN 26087 --- [ocp-async-3,4c84684e189e5d62,dc71bc4d1590177a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:02.600 INFO 26087 --- [ocp-async-1,8464d8355dd11235,c6a028de886c0acf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:03.271 INFO 26087 --- [prometheus-minute-schedule-1,ef869770bb08a68b,04bff646c33b7513] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:04.600 WARN 26087 --- [ocp-async-9,d9dbeefaf9b3bd62,d0cd3e42fea409ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:04.600 INFO 26087 --- [ocp-async-7,df84eb75a8fde650,ff6bb9abc8c5427b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:05.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:05.477 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:06.271 INFO 26087 --- [prometheus-minute-schedule-1,3d7001fa0b765acd,f4899b889bcd4369] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:06.272 INFO 26087 --- [prometheus-cache-monitor1,3c5ad091cdb0dded,0dec820c3cf2b9fb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:34:06.272 INFO 26087 --- [prometheus-cache-monitor1,3c5ad091cdb0dded,0dec820c3cf2b9fb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:34:06.283 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:28:06.328, end=2025-04-08T21:33:06.279} 2025-04-08 21:34:06.284 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:28:06.332, end=2025-04-08T21:33:06.28} 2025-04-08 21:34:06.287 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:34:06.287 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:34:06.287 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:34:06.287 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:34:06.483 INFO 26087 --- [ocp-async-7,26819d38c507d9d3,4aef1d3d43f0a41d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:34:06.580 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:30:06.533, end=2025-04-08T21:32:06.578} 2025-04-08 21:34:06.582 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:34:06.582 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:34:06.600 WARN 26087 --- [ocp-async-10,fec5e4767416c8ba,768ad8e3f1e5694c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:06.600 INFO 26087 --- [ocp-async-2,ffe5e025fb6f753d,794f23d14a960fb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:34:06.622919+08:00, lastProcessTime=2025-04-08T21:33:56.623445+08:00 2025-04-08 21:34:06.694 INFO 26087 --- [ocp-schedules-24,e70f81b42664dd38,1f9fc058a54cbbbe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:34:06.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:34:06.622919+08:00 2025-04-08 21:34:08.478 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:08.478 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:08.580 INFO 26087 --- [ocp-schedules-24,99aad4acd1c77d76,689b34005a4b4cf2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:34:08.581 INFO 26087 --- [ocp-schedules-24,99aad4acd1c77d76,689b34005a4b4cf2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:34:08.600 INFO 26087 --- [ocp-async-5,b2793a6a108ed020,aa92334fd8d1a3a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:08.600 WARN 26087 --- [ocp-async-6,ae006bc4b6f96115,176dde351930c889] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:09.271 INFO 26087 --- [prometheus-minute-schedule-1,f21b5f8795ee5b09,73e7e06f0ab460d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:10.001 INFO 26087 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:34:10.001 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:34:10.520 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:34:10.600 WARN 26087 --- [ocp-async-1,b4319446e7f5e189,9ef084bbfee8c514] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:10.600 INFO 26087 --- [ocp-async-9,5da98c9b080aec0d,e20340f62372665c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:11.478 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:11.479 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:11.510 INFO 26087 --- [ocp-schedules-12,0d703206796bcb2a,de48f72d601e7ab2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:34:12.271 INFO 26087 --- [prometheus-minute-schedule-1,c1816aa4548ab9af,50d5de17f1f1cea1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:12.414 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119180, nodeCount=102, affectRows=102 2025-04-08 21:34:12.415 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119180, nodeCount=102, validCount=102 2025-04-08 21:34:12.415 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119180, nodeCount=276, affectRows=276 2025-04-08 21:34:12.418 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119180, nodeCount=275, validCount=275 2025-04-08 21:34:12.601 WARN 26087 --- [ocp-async-9,b7736552e78cd807,5f1d7651dc6f796f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:12.601 INFO 26087 --- [ocp-async-8,5c02d63378a7e7f1,007881f383256cd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:14.479 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:14.480 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:14.601 INFO 26087 --- [ocp-async-6,d85ba038eb114205,3ffa3756ba38a03f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:14.601 WARN 26087 --- [ocp-async-2,7d6f82cb3dde07cb,b133ac85967578dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:15.051 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e475f5daaeeb78db,a5981dacc1df43c2] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/profiles/me/credentials/validate, client 10.11.12.102, traceId e475f5daaeeb78db] 2025-04-08 21:34:15.053 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e475f5daaeeb78db,a5981dacc1df43c2] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/profiles/me/credentials/validate?null, client=10.11.12.102, traceId=e475f5daaeeb78db, method=SuccessResponse com.oceanbase.ocp.server.common.controller.CredentialController.validateConnect(ValidateCredentialParam), args=ValidateCredentialParam(targetType=HOST, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=null, batchSshValidateProperty=BatchSshValidateProperty(addressPairs=[BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)], type=null, username=null, passphrase=xxx allowSudo=true, changepassphrase=xxx credentialId=1, localPrivateKeyPath=null)),] 2025-04-08 21:34:15.056 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e475f5daaeeb78db,a5981dacc1df43c2] c.o.o.s.c.CredentialServiceImpl : Batch validate credential SshSecret(type=PASSWORD, allowSudo=true, username=root, hosts=[], localPrivateKeyPath=null) for connecting hosts: [BatchSshValidateProperty.AddressPair(hostAddress=192.168.1.61, sshPort=22, vpcId=1)] 2025-04-08 21:34:15.057 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:15.072 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.072 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:15.086 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.086 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:15.101 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.101 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo 1 on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.101 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:15.116 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.116 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo -n true on 192.168.1.61 2025-04-08 21:34:15.142 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo -n true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.142 INFO 26087 --- [pool-28-thread-1,e475f5daaeeb78db,fec70946b1705ad6] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo -n true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo -n true, out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:15.142 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,e475f5daaeeb78db,a5981dacc1df43c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/profiles/me/credentials/validate client=10.11.12.102, traceId=e475f5daaeeb78db, duration=93 ms] 2025-04-08 21:34:15.207 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/takeOver, client 10.11.12.102, traceId 29e4ae1f6d16bfd8] 2025-04-08 21:34:15.209 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/takeOver?null, client=10.11.12.102, traceId=29e4ae1f6d16bfd8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster(TakeOverClusterParam), args=TakeOverClusterParam(switchConfigUrl=true, hostInfo=TakeOverClusterParam.HostInfo(kind=DEDICATED_PHYSICAL_MACHINE, publishPorts=null, hostTypeId=1, sshPort=22, credentialId=1)),] 2025-04-08 21:34:15.216 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=ob-cluster-192.168.1.61#2881, method=takeOverCluster 2025-04-08 21:34:15.219 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.220 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.220 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.221 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.221 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.221 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.222 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.222 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.223 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.223 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.224 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.224 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-04-08 21:34:15.229 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.229 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.230 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.230 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster_id] 2025-04-08 21:34:15.235 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.235 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.236 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.236 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-04-08 21:34:15.242 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.242 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.243 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.243 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.243 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.244 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-04-08 21:34:15.244 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.244 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.245 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.245 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.245 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.246 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [data_dir] 2025-04-08 21:34:15.251 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.251 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.251 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.252 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-04-08 21:34:15.252 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.252 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.253 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.255 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-04-08 21:34:15.256 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.256 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.257 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.257 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.257 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:34:15.258 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.261 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-04-08 21:34:15.271 INFO 26087 --- [prometheus-minute-schedule-1,7d8179aefe33f1b7,874ad375864e1340] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:15.272 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.272 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.273 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.273 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:34:15.288 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.288 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.289 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.290 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.290 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.291 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.291 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.292 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:34:15.309 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.309 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.310 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.310 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-04-08 21:34:15.311 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.311 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.312 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.312 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:34:15.331 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.331 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.332 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.332 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.333 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.333 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.333 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.334 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.334 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.334 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:34:15.349 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.350 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.351 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.351 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-04-08 21:34:15.352 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.352 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.353 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.353 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:34:15.369 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.369 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.370 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.371 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.371 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.372 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.372 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.372 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.373 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.373 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:34:15.389 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.390 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.391 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.391 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-04-08 21:34:15.394 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.394 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.395 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.395 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.396 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.396 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.397 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.397 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(1) FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'ocp' AND TABLE_NAME = 'ops_config' 2025-04-08 21:34:15.406 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.407 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.407 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.408 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.409 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.409 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:34:15.409 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.410 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:34:15.410 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.410 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:34:15.414 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.415 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.415 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.416 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.416 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.416 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.417 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.417 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.418 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.418 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:34:15.426 ERROR 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.compute.host.ip.not.found, and args [class com.oceanbase.ocp.core.metadb.repo.compute.host.entity.HostEntity, 192.168.1.61] 2025-04-08 21:34:15.431 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.host.service.RegionServiceImpl : Created region: 1000004 deault_region 2025-04-08 21:34:15.435 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.host.service.IdcServiceImpl : Created idc: 1000003 default_idc 2025-04-08 21:34:15.448 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000006, resourceId=1000006 2025-04-08 21:34:15.455 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.s.o.OrganizationResourceManager : add resource for current organization,organizationId=10000000,resourceType=1000003, resourceId=1000003 2025-04-08 21:34:15.462 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000003, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:34:15.469 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000003, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:34:15.469 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000003, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:34:15.470 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.CredentialServiceImpl : Credential clusterName=ocpdb|tenantName=sys|username=root is already exists 2025-04-08 21:34:15.470 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.CredentialServiceImpl : Delete credential:clusterName=ocpdb|tenantName=sys|username=root 2025-04-08 21:34:15.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=OB, validate=false, name=null, description=null, obJdbcCredentialProperty=ObJdbcCredentialProperty(clusterId=1000003, tenantId=null, clusterName=ocpdb, tenantName=sys, username=root), sshCredentialProperty=null, agentCredentialProperty=null) 2025-04-08 21:34:15.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.472 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:34:15.473 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.473 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:34:15.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.474 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:34:15.478 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.479 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.479 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.480 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.480 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.481 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.481 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.481 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:15.482 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:15.482 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-04-08 21:34:15.487 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.o.i.c.ClusterOperationServiceImpl : meta cluster detected. 2025-04-08 21:34:15.496 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.event.CmdbSyncEventPublisher : [CMDB]:Publish cmdb sync event:UPDATE APP GROUP, property:UpdateAppGroupProperty(serialNumbers=[null], appGroup=null) 2025-04-08 21:34:15.497 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.distributed.event.EventUpdater : [OCP116301] distributed event published by b88d8e1c7d, eventId=1000013, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:34:15.498 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Take over primary OB cluster, nodeCount=39, taskCount=39 2025-04-08 21:34:15.498 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000003, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, createTime=null, startTime=2025-04-08T21:34:15.498+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update server process info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.server.UpdateObServerInfoTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=26, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Prepare take over cluster, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.PrepareTakeOverClusterTask, seriesId=15, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Sync all tenant information, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoTask, seriesId=18, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=32, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, 2025-04-08 21:34:15.528 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000147} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.534 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000138} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.536 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000125} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1000003, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000006, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.537 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000161} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.538 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000151} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.539 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000140} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.540 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000152} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.540 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000150} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.541 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000160} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.542 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000153} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1000003, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=1000006, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.542 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000158} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=1000006, svrIp='192.168.1.61', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.543 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=1000136} BaseTagEntity{id=null, clusterName='ocpdb', clusterId=1000003, obClusterId=1744114127, zoneName='zone1', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=null, svrIp='null', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-04-08 21:34:15.544 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=1000016, taskName=Take over primary OB cluster, argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}} 2025-04-08 21:34:15.586 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/takeOver client=10.11.12.102, traceId=29e4ae1f6d16bfd8, duration=381 ms] 2025-04-08 21:34:15.586 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,29e4ae1f6d16bfd8,035c782cd2e6654e] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=TAKE_OVER_OB_CLUSTER, eventType=OB_CLUSTER_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObClusterController.takeOverObCluster, resourceType=OB_CLUSTER, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=192.168.1.61:2881, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=1000016, eventInfo={"param":{"address":"192.168.1.61","connectionMode":"direct","hostInfo":{"credentialId":1,"hostTypeId":1,"kind":"DEDICATED_PHYSICAL_MACHINE","sshPort":22},"port":2881,"rootSysPassword":"******","saveToCredential":true,"switchConfigUrl":true}}, executionType=NEW, requestUrl=http://192.168.1.61:8080/api/v2/ob/clusters/takeOver, clientIp=10.11.12.102, serverIp=192.168.1.61, startTime=2025-04-08T21:34:15.209+08:00, endTime=null, role=null, aliUid=null) 2025-04-08 21:34:15.599 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,72cee995bb68c633] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000127, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:15.653 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,b73b372a7628046e] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:15.739 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster [ocpdb] cache 2025-04-08 21:34:15.740 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh cluster cache success 2025-04-08 21:34:15.740 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db 2025-04-08 21:34:15.741 INFO 26087 --- [ForkJoinPool.commonPool-worker-12,,] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Get ob cluster ocpdb cache info from db success. 2025-04-08 21:34:15.735 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,b73b372a7628046e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000127, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, service_name=ocpdb:1744114127, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, latest_execution_start_time=2025-04-08T21:34:15.594+08:00, sub_task_instance_name=Prepare take over cluster, sub_task_instance_id=1000127}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:15.785 INFO 26087 --- [ocp-updater-0,,] c.o.o.c.distributed.event.EventUpdater : [OCP116302] distributed event received by b88d8e1c7d, eventId=1000013, event=RefreshClusterCacheEvent{name=refresh-config-server-cache, payload=["ocpdb"]} 2025-04-08 21:34:15.810 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,b73b372a7628046e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000127, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:15.812 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,b73b372a7628046e] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000150] 2025-04-08 21:34:16.433 INFO 26087 --- [ocp-schedules-23,84f41966c8a93595,3ba596361ffe0ea6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:34:16.434 INFO 26087 --- [ocp-schedules-12,dc74926a6d36a98d,e5bcd030d35a9d6b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:34:16.436 INFO 26087 --- [ocp-schedules-23,84f41966c8a93595,3ba596361ffe0ea6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:34:16.437 INFO 26087 --- [ocp-schedules-11,11e84eb6697d2884,c53e8b2a7069ed31] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:34:16.437 INFO 26087 --- [ocp-schedules-11,11e84eb6697d2884,c53e8b2a7069ed31] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:34:16.437 INFO 26087 --- [ocp-schedules-10,793e3b6503983e32,44a838fa948597fa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:34:16.439 INFO 26087 --- [ocp-schedules-11,291bfe217fb661ca,9829953519b86407] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:34:16.439 INFO 26087 --- [ocp-schedules-11,291bfe217fb661ca,9829953519b86407] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:34:16.439 INFO 26087 --- [ocp-schedules-2,063d397190c49f6e,fa6a04c99aaeafcb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:34:16.439 INFO 26087 --- [ocp-schedules-2,063d397190c49f6e,fa6a04c99aaeafcb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-15,ba543ae3d03caf1d,ff83566cff948e12] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-12,dc74926a6d36a98d,e5bcd030d35a9d6b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-1,7619fe8a8e8c9a13,2324ea364a46c99f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-12,7ab40a824d0ee063,dc8b118431be6683] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-1,7619fe8a8e8c9a13,2324ea364a46c99f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-12,7ab40a824d0ee063,dc8b118431be6683] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:16.441 INFO 26087 --- [ocp-schedules-15,ba543ae3d03caf1d,ff83566cff948e12] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:34:16.447 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:34:16.447 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:34:16.455 INFO 26087 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.distributed.event.EventUpdater : [OCP116303] distributed event wait completed, eventId=1000013 2025-04-08 21:34:16.463 INFO 26087 --- [ocp-schedules-8,0c5986a1224087c1,dc2e9a6b929c35db] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:34:16.478 INFO 26087 --- [ocp-schedules-22,497de1bfc08cdecd,3c43b2095ffebc61] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 21:34:16.478 INFO 26087 --- [ocp-schedules-22,497de1bfc08cdecd,3c43b2095ffebc61] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 21:34:16.480 INFO 26087 --- [ocp-schedules-27,24564b74128b0e01,7fe6bbf228636650] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:34:16.481 INFO 26087 --- [ocp-schedules-27,24564b74128b0e01,7fe6bbf228636650] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:34:16.483 INFO 26087 --- [ocp-async-1,92efd47766306e98,10e197bae2e18bdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:34:16.528 INFO 26087 --- [ocp-schedules-24,942c4af952e10427,7f70269d5b69271c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:34:16.563 INFO 26087 --- [ocp-schedules-29,9ce9a82e09c94f03,92e37a4100e4164b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:34:16.570 INFO 26087 --- [ocp-schedules-20,e8f888d7710a4057,4b3ace7e0db2d03e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:34:16.570 INFO 26087 --- [ocp-schedules-20,e8f888d7710a4057,4b3ace7e0db2d03e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:34:16.580 INFO 26087 --- [ocp-schedules-2,23b275c5fe57ef89,24ae8e2c19cd493f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:34:16.580 INFO 26087 --- [ocp-schedules-2,23b275c5fe57ef89,24ae8e2c19cd493f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:34:16.581 INFO 26087 --- [ocp-schedules-18,f2470ba1a7c4d1ed,08e462241e89e79f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:34:16.581+08:00, clusters=[1000003] 2025-04-08 21:34:16.582 INFO 26087 --- [ocp-schedules-18,f2470ba1a7c4d1ed,08e462241e89e79f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:34:16.583 INFO 26087 --- [ocp-schedules-10,605d3e512e928a69,a820c17b6289cd4f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:34:16.583 INFO 26087 --- [ocp-schedules-11,0c37cb4918933984,29fbed03c65241e3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:34:16.583+08:00, clusters=[1000003] 2025-04-08 21:34:16.585 INFO 26087 --- [ocp-schedules-11,0c37cb4918933984,29fbed03c65241e3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:34:16.601 WARN 26087 --- [ocp-async-9,119e617314b15207,d4913c0bd1e9d931] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:16.601 INFO 26087 --- [ocp-async-10,0457130fe223f17a,23f710e6609233ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:16.604 INFO 26087 --- [ocp-schedules-23,cc326d6548e7ad11,ddcceaedbffb4e41] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:34:16.604 INFO 26087 --- [ocp-schedules-23,cc326d6548e7ad11,ddcceaedbffb4e41] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:34:16.607 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,8d9f6c0775801de1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000150, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:16.607 INFO 26087 --- [ocp-schedules-7,a9abe1c5e1be86df,dce6fe47e64a0a0c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:34:16.608 INFO 26087 --- [ocp-schedules-23,cc326d6548e7ad11,ddcceaedbffb4e41] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:34:16.608 INFO 26087 --- [ocp-schedules-23,cc326d6548e7ad11,ddcceaedbffb4e41] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:34:16.609 INFO 26087 --- [ocp-schedules-7,a9abe1c5e1be86df,dce6fe47e64a0a0c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:34:16.609 INFO 26087 --- [ocp-schedules-7,a9abe1c5e1be86df,dce6fe47e64a0a0c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:34:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:34:16.622739+08:00, lastProcessTime=2025-04-08T21:34:06.622919+08:00 2025-04-08 21:34:16.694 INFO 26087 --- [ocp-schedules-27,0f66e9626a52f89f,22ec3ae2afa5666f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:34:16.693 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:16.697 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000150, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:16.605+08:00, sub_task_instance_name=Pre check for create host, sub_task_instance_id=1000150}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:16.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build alarmSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 21:34:16.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:34:16.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : build recoverSubscribedJudger, currentSubscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin-ocpdb, subscribedJudgerVersion subscription-0-2025-04-08T21:02:45.218+08:00_groups-11-2025-04-08T20:19:17+08:00-2025-04-08T20:19:17+08:00_detectRules-544-2025-04-08T12:19:42.887240Z_readableClusters_1_admin- 2025-04-08 21:34:16.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.distribute.DistributeBuilder : buildRecoverMessageSubscribedJudger, usersSubscribedCount=0, userEnableReceiveCount=0 2025-04-08 21:34:16.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.a.c.d.MessageDistributorFactory : buildAllDistributors 2025-04-08 21:34:16.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:34:16.622739+08:00 2025-04-08 21:34:16.712 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:16.729 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.731 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.s.t.b.host.PreCreateHostCheckTask : Pre check for create host, ip=192.168.1.61 2025-04-08 21:34:16.755 INFO 26087 --- [ocp-schedules-6,23c391e7bce8da93,c9898daf28b67935] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:34:16.756 INFO 26087 --- [ocp-schedules-6,23c391e7bce8da93,c9898daf28b67935] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:16.769 INFO 26087 --- [ocp-schedules-14,364e9a308bf2aa52,f0762b1ac392fcfe] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:34:16.770 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:16.770 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-04-08 21:34:16.770 INFO 26087 --- [ocp-schedules-14,364e9a308bf2aa52,f0762b1ac392fcfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:16.771 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:34:16.771 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:34:16.770 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:16.785 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:34:16.786 ERROR 26087 --- [ocp-schedules-14,364e9a308bf2aa52,f0762b1ac392fcfe] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:34:16.786 INFO 26087 --- [ocp-schedules-14,364e9a308bf2aa52,f0762b1ac392fcfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:34:16.798 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.805 INFO 26087 --- [ocp-schedules-28,50502345c10e8264,4b16b13aef745609] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:34:16.805 INFO 26087 --- [ocp-schedules-28,50502345c10e8264,4b16b13aef745609] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:34:16.801 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo '' on 192.168.1.61 2025-04-08 21:34:16.808 INFO 26087 --- [ocp-schedules-28,50502345c10e8264,4b16b13aef745609] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:34:16.808 INFO 26087 --- [ocp-schedules-28,50502345c10e8264,4b16b13aef745609] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:34:16.811 INFO 26087 --- [ocp-schedules-28,50502345c10e8264,4b16b13aef745609] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:34:16.811 INFO 26087 --- [ocp-schedules-28,50502345c10e8264,4b16b13aef745609] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:34:16.820 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo '' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.823 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.e.internal.template.SshTemplate : SSH execute end: echo '' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=echo '', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.828 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh console output, expected output is 'true', real output is 'true' 2025-04-08 21:34:16.888 INFO 26087 --- [ocp-schedules-19,4a4d3604e7755f9e,d573bb3bea9bb75b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:34:16.842 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, ip=192.168.1.61, port=22, loopCount=1, minSuccessCount=1 2025-04-08 21:34:16.889 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:16.906 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.911 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: whoami on 192.168.1.61 2025-04-08 21:34:16.934 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: whoami on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.939 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.e.internal.template.SshTemplate : SSH execute end: whoami on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=whoami, out=root, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:16.946 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.executor.internal.util.PingUtils : Rtt by ssh, rawList=[57] 2025-04-08 21:34:16.960 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check ssh ping delay, delayMillis=57, maxAcceptableDelayMillis=3000 2025-04-08 21:34:16.982 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[192.168.1.61], timeoutMillis=20000 2025-04-08 21:34:16.989 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1744119256 0 0 , error= 2025-04-08 21:34:16.991 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.s.t.b.host.PreCreateHostCheckTask : Check clock diff by ssh, clockDiff=0, maxDiff=50 2025-04-08 21:34:17.001 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000150, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:17.003 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,50009794cb726186] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000162] 2025-04-08 21:34:17.480 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:17.480 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:17.510 INFO 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:34:17.510 INFO 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:34:17.513 INFO 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:34:17.514 WARN 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:34:17.514 INFO 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:34:17.515 WARN 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:34:17.518 WARN 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:34:17.518 ERROR 26087 --- [ocp-schedules-25,0d0b020746f2d3d1,7e550ba1cd50858b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=COMPUTE_HOST_AGENT_CREDENTIAL_NOT_FOUND, args=1000006 at com.oceanbase.ocp.core.i18n.ErrorCodes$Kind$8.exception(ErrorCodes.java:1788) at com.oceanbase.ocp.core.i18n.ErrorCodes.exception(ErrorCodes.java:1669) at com.oceanbase.ocp.core.util.ExceptionUtils.require(ExceptionUtils.java:154) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.getAgentCredential(ExecutorPropertiesBuilderImpl.java:378) at com.oceanbase.ocp.compute.host.executor.ExecutorPropertiesBuilderImpl.buildMonitorExecutorProperties(ExecutorPropertiesBuilderImpl.java:184) at com.oceanbase.ocp.compute.host.executor.RemoteExecutorFactoryImpl.createMonitorAgentExecutor(RemoteExecutorFactoryImpl.java:99) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:61) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted 2025-04-08 21:34:17.624 INFO 26087 --- [ocp-schedules-5,8e67fd73c7a54355,0a8b50e1c497adeb] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,,,,,C_sgkydb-1744016068-PRIMARY,C_sgkydb-1744016068-PRIMARY/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65,C_sgkydb-1744016068/H_192.168.1.62,H_192.168.1.63,H_192.168.1.64,H_192.168.1.65, current targets ,,,,,C_ocpdb-1744114127-PRIMARY,C_ocpdb-1744114127-PRIMARY/H_192.168.1.61,C_ocpdb-1744114127/H_192.168.1.61,H_192.168.1.61,H_192.168.1.61 2025-04-08 21:34:17.642 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,e30e74162dfe7d85] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000162, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:17.668 INFO 26087 --- [ocp-schedules-23,20731a0927bf49c8,463a93010f2a4c8b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:34:17.653 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,e044d7726c6418cb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:17.741 INFO 26087 --- [ocp-schedules-23,20731a0927bf49c8,463a93010f2a4c8b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:34:17.741 INFO 26087 --- [ocp-schedules-23,20731a0927bf49c8,463a93010f2a4c8b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:34:17.698 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,e044d7726c6418cb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000162, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:17.640+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000162}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:17.747 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,e044d7726c6418cb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000162, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:17.748 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,e044d7726c6418cb] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000158] 2025-04-08 21:34:18.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,581839158eb478b0,4a8529aff7168443] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=581839158eb478b0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:18.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,10f4ab387982407f,1cd0502b7499bb92] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.11.12.102, traceId=10f4ab387982407f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-04-08 21:34:18.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,a4d1eb8ec3b98b97,19a8ce309b35594e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.11.12.102, traceId=a4d1eb8ec3b98b97, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-04-08 21:34:18.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,a4d1eb8ec3b98b97,19a8ce309b35594e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.11.12.102, traceId=a4d1eb8ec3b98b97, duration=3 ms] 2025-04-08 21:34:18.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,22708b8e44b542de,98f6737b0663559c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=22708b8e44b542de, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:34:18.076 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,22708b8e44b542de,98f6737b0663559c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=22708b8e44b542de, duration=4 ms] 2025-04-08 21:34:18.077 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,9a55dd94810cec9b,1d3229f85defca27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=9a55dd94810cec9b, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:34:18.079 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,10f4ab387982407f,1cd0502b7499bb92] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.11.12.102, traceId=10f4ab387982407f, duration=7 ms] 2025-04-08 21:34:18.085 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,581839158eb478b0,4a8529aff7168443] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=581839158eb478b0, duration=13 ms] 2025-04-08 21:34:18.131 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,619d88b919080853,849a51cf2bd32d1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.11.12.102, traceId=619d88b919080853, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-04-08 21:34:18.137 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,619d88b919080853,849a51cf2bd32d1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.11.12.102, traceId=619d88b919080853, duration=8 ms] 2025-04-08 21:34:18.171 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,9a55dd94810cec9b,1d3229f85defca27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=9a55dd94810cec9b, duration=99 ms] 2025-04-08 21:34:18.221 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:34:18.221 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-8,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:34:18.221 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:34:18.271 INFO 26087 --- [prometheus-minute-schedule-1,e41516059878f134,2a6ee8019d09e10d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-04-08 21:34:18.368 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-04-08 21:34:18.369 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-04-08 21:34:18.370 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-04-08 21:34:18.371 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 0 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 0 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-04-08 21:34:18.372 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 0 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-04-08 21:34:18.373 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 0 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-04-08 21:34:18.374 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 0 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-04-08 21:34:18.375 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-04-08 21:34:18.376 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-04-08 21:34:18.377 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 0 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-04-08 21:34:18.378 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-04-08 21:34:18.392 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-04-08 21:34:18.393 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-04-08 21:34:18.394 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-04-08 21:34:18.395 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-04-08 21:34:18.396 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-04-08 21:34:18.397 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-04-08 21:34:18.398 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY/H_192.168.1.61 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-04-08 21:34:18.399 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_192.168.1.61 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_ocpdb-1744114127-PRIMARY 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-04-08 21:34:18.400 INFO 26087 --- [ocp-schedules-4,f496058a92b22c35,7c24123b6ae0bfae] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=132, updatedCount=132, failedToUpdateCount=0 2025-04-08 21:34:18.587 INFO 26087 --- [ocp-schedules-9,bd0574262c4cf4e2,c86d1d950b338221] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:34:18.588 INFO 26087 --- [ocp-schedules-9,bd0574262c4cf4e2,c86d1d950b338221] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:34:18.601 WARN 26087 --- [ocp-async-2,0ef419daad31975e,5d3b38a0b16dc007] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:18.601 INFO 26087 --- [ocp-async-6,250b884e778f4e79,ac3eca4957fb5575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:18.652 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,042eed5da05c9b88] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000158, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:18.659 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,4a5ae2e6e02783c5,00dae75fa979bfaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016/subtasks/1000158/logSegments?taskInstanceId=1000016&subtaskInstanceId=1000158, client=10.11.12.102, traceId=4a5ae2e6e02783c5, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000016,1000158,] 2025-04-08 21:34:18.660 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,4a5ae2e6e02783c5,00dae75fa979bfaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016/subtasks/1000158/logSegments client=10.11.12.102, traceId=4a5ae2e6e02783c5, duration=4 ms] 2025-04-08 21:34:18.734 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:18.781 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000158, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:18.650+08:00, sub_task_instance_name=Update host arch info, sub_task_instance_id=1000158}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:18.783 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Begin to update host arch info 2025-04-08 21:34:18.808 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:18.893 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:18.952 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:18.978 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:19.019 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... uname_ext on 192.168.1.61 2025-04-08 21:34:19.049 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... uname_ext on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:19.065 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... uname_ext on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... uname_ext, out={"sysname": "Linux", "nodename": "localhost.localdomain", "release": "5.10.134-18.an8.x86_64", "version": "#1 SMP Fri Dec 13 16:32:58 CST 2024", "machine": "x86_64", "processor": "x86_64", "is_in_docker": false, "linux_distribution_name": "anolis", "linux_distribution_release": "8.9"}, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:19.087 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] c.o.o.s.t.b.h.UpdateHostSystemInfoTask : Success to update host arch info 2025-04-08 21:34:19.097 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000158, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:19.099 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,00ed34ff7fc83cb4] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000144] 2025-04-08 21:34:19.476 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,afaf6236e1b6545a,c101e9c69458c067] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=afaf6236e1b6545a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:19.486 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,afaf6236e1b6545a,c101e9c69458c067] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=afaf6236e1b6545a, duration=13 ms] 2025-04-08 21:34:19.685 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,8e533d1050217292] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000144, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:19.695 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,f33995ae39ace1fc] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:19.702 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,f33995ae39ace1fc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000144, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:19.684+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000144}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:19.714 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,1c4c92168236b44b,a3bca1a95436b7e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016/subtasks/1000158/logSegments?taskInstanceId=1000016&subtaskInstanceId=1000158, client=10.11.12.102, traceId=1c4c92168236b44b, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000016,1000158,] 2025-04-08 21:34:19.716 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,1c4c92168236b44b,a3bca1a95436b7e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016/subtasks/1000158/logSegments client=10.11.12.102, traceId=1c4c92168236b44b, duration=6 ms] 2025-04-08 21:34:19.723 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,f33995ae39ace1fc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000144, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:19.724 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,f33995ae39ace1fc] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000138] 2025-04-08 21:34:20.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,5ff0c45dec46dcb4,0b993d17c15af4e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.11.12.102, traceId=5ff0c45dec46dcb4, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-04-08 21:34:20.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,10af876927230ac2,d8fa6dc328f77916] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.11.12.102, traceId=10af876927230ac2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-04-08 21:34:20.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,5ff0c45dec46dcb4,0b993d17c15af4e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.11.12.102, traceId=5ff0c45dec46dcb4, duration=2 ms] 2025-04-08 21:34:20.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,10af876927230ac2,d8fa6dc328f77916] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.11.12.102, traceId=10af876927230ac2, duration=2 ms] 2025-04-08 21:34:20.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,b750aa3f376a9467,b2748d279c1947d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.11.12.102, traceId=b750aa3f376a9467, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-04-08 21:34:20.075 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,b750aa3f376a9467,b2748d279c1947d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.11.12.102, traceId=b750aa3f376a9467, duration=2 ms] 2025-04-08 21:34:20.091 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.11.12.102, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-04-08 21:34:20.092 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.11.12.102, client=GET, traceId=] 2025-04-08 21:34:20.092 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.11.12.102, traceId=, duration=0 ms] 2025-04-08 21:34:20.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:20.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:34:20.601 WARN 26087 --- [ocp-async-5,e9c8a09be72e0646,f09e2751a0c2d5b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:20.601 INFO 26087 --- [ocp-async-1,7b8d3d5cefaa2072,16eb01c1f84deaeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:20.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,982b9e802e357d24,997a7177a847ebce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=982b9e802e357d24, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:20.679 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,982b9e802e357d24,997a7177a847ebce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=982b9e802e357d24, duration=12 ms] 2025-04-08 21:34:20.693 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,dcee0b3757af7a51] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000138, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:20.701 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:20.704 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000138, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:20.691+08:00, sub_task_instance_name=Uninstall legacy obagent, sub_task_instance_id=1000138}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:20.713 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent begin 2025-04-08 21:34:20.748 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:20.779 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:20.819 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:20.838 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:20.848 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:20.890 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:20.942 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_agentd|obstat|ob_sql|ob_sql_audit|ob_logtailer|ob_cleaner|ob_exporter_1|ob_exporter_60|node_exporter_wrapper_ob|node_exporter_wrapper_obproxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:20.949 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:20.969 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:20.976 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61 2025-04-08 21:34:21.271 INFO 26087 --- [prometheus-minute-schedule-1,1e6ff282df02ad30,b6cff3926d52caec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:21.348 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:21.365 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-obagent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-obagent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:21.376 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-obagent 2025-04-08 21:34:21.379 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-obagent 2025-04-08 21:34:21.389 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:21.410 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:21.424 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61 2025-04-08 21:34:21.529 INFO 26087 --- [ocp-schedules-5,304756371408ff31,42521425e7e6e2be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:34:21.798 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:21.805 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package '^t-oceanbase-ob-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package '^t-oceanbase-ob-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:21.807 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.ocp.executor.executor.SshExecutor : package name not found from host:^t-oceanbase-ob-agent 2025-04-08 21:34:21.809 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=^t-oceanbase-ob-agent 2025-04-08 21:34:21.811 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.s.t.b.o.UninstallLegacyObAgentTask : [UninstallLegacyObAgentTask] uninstall legacy obagent success 2025-04-08 21:34:21.816 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000138, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:21.817 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,fe09c3df493b21bb] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000126] 2025-04-08 21:34:21.868 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,4e61459cf877c5c6,8609130700be8c64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=4e61459cf877c5c6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:21.878 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,4e61459cf877c5c6,8609130700be8c64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=4e61459cf877c5c6, duration=13 ms] 2025-04-08 21:34:22.601 WARN 26087 --- [ocp-async-4,3cb61dabf70f1957,dcc7236b3ad22612] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:22.601 INFO 26087 --- [ocp-async-7,4d882ad91df399ad,62187e1988aa1e4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:22.702 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,efd48087a854030e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000126, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:22.787 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,a9c99a6d853d8935] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:22.834 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,a9c99a6d853d8935] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000126, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:22.701+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000126}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:22.842 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,a9c99a6d853d8935] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000126, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:22.843 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,a9c99a6d853d8935] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000151] 2025-04-08 21:34:23.051 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,8b5ecda5f12f6134,7450a911254afa10] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=8b5ecda5f12f6134, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:23.061 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,8b5ecda5f12f6134,7450a911254afa10] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=8b5ecda5f12f6134, duration=14 ms] 2025-04-08 21:34:23.086 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,90bca011e3fb0dec,485c13b67f53b9ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.11.12.102, traceId=90bca011e3fb0dec, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-04-08 21:34:23.088 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-5,90bca011e3fb0dec,485c13b67f53b9ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.11.12.102, traceId=90bca011e3fb0dec, duration=5 ms] 2025-04-08 21:34:23.481 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:23.482 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:23.732 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,40da7d23fb295ca6] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000151, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:23.741 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:23.745 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000151, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:23.730+08:00, sub_task_instance_name=Uninstall legacy ocp agent, sub_task_instance_id=1000151}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:23.749 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] start legacy ocp agent uninstallation 2025-04-08 21:34:23.756 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:23.779 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.789 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:23.809 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.850 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:23.879 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.886 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'supervisord' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.890 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:23.907 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.948 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:23.974 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.990 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:23.995 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.012 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.053 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.091 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.095 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'pos_proxy' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.101 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.121 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.161 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.187 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.189 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'node_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.195 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.214 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.263 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,a3bfc82eb556c119,16f01364a60bf077] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=a3bfc82eb556c119, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:24.254 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.271 INFO 26087 --- [prometheus-minute-schedule-1,2a6a8b4248170cf6,49ec8d262ee3fc84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:24.273 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,a3bfc82eb556c119,16f01364a60bf077] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=a3bfc82eb556c119, duration=14 ms] 2025-04-08 21:34:24.292 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.295 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.297 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.316 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.357 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.389 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.436 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'obstat2' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.439 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.456 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.496 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.565 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.591 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_logtailer' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.596 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.601 WARN 26087 --- [ocp-async-10,1731ac49a4a033c4,da01cfce599fb771] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:24.601 INFO 26087 --- [ocp-async-2,baaefc0a4b90422a,a5dd42979e68e401] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:24.618 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.658 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.694 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.739 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ob_cleaner' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.752 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.781 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.821 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:24.847 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.851 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_exporter' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.854 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:24.871 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:24.911 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:34:25.312 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.317 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.325 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:34:25.327 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:34:25.330 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:25.367 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.408 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:25.434 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.440 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_agentd' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.441 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:25.465 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,90bf7c93c64b25c9,9a87065f44620678] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=90bf7c93c64b25c9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:25.473 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-7,90bf7c93c64b25c9,9a87065f44620678] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=90bf7c93c64b25c9, duration=12 ms] 2025-04-08 21:34:25.458 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.499 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:25.529 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.543 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_mgragent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.552 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:25.574 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.614 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61 2025-04-08 21:34:25.692 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.774 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: (pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=(pgrep -af 'ocp_monagent' | grep -cv pgrep | xargs --no-run-if-empty) || true, out=0, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.791 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.s.c.AgentInstallationTaskService : Uninstall compatibility history ocp-agent, rpmName=t-oceanbase-ocp-agent 2025-04-08 21:34:25.796 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:25.814 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:25.855 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61 2025-04-08 21:34:26.284 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.291 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 't-oceanbase-ocp-agent' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 't-oceanbase-ocp-agent', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.294 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.executor.executor.SshExecutor : package name not found from host:t-oceanbase-ocp-agent 2025-04-08 21:34:26.296 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=t-oceanbase-ocp-agent 2025-04-08 21:34:26.298 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.s.c.AgentInstallationTaskService : Uninstall ocp-agent, rpmName=ocp-agent-ce 2025-04-08 21:34:26.303 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:26.323 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.364 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61 2025-04-08 21:34:26.442 INFO 26087 --- [ocp-schedules-12,7ab40a824d0ee063,dc8b118431be6683] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:34:26.442 INFO 26087 --- [ocp-schedules-1,7619fe8a8e8c9a13,2324ea364a46c99f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:34:26.482 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:26.483 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:26.483 INFO 26087 --- [ocp-async-8,1f32e0639a56b058,d2f9e35d8f84cd06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:34:26.602 INFO 26087 --- [ocp-async-6,c09af8d8cae9ddd6,70c7661e1bb63968] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:26.602 WARN 26087 --- [ocp-async-4,e638061594baace9,07c5b67b55ff3bff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:34:26.622976+08:00, lastProcessTime=2025-04-08T21:34:16.622739+08:00 2025-04-08 21:34:26.659 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,8dc3738baa700b39,097edb25b7645d74] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=8dc3738baa700b39, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:26.668 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,8dc3738baa700b39,097edb25b7645d74] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=8dc3738baa700b39, duration=12 ms] 2025-04-08 21:34:26.696 INFO 26087 --- [ocp-schedules-12,1ca20e74cffcdb43,d53c4035f7b7b111] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:34:26.703 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:34:26.622976+08:00 2025-04-08 21:34:26.747 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.753 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: #!/bin/bash ... find_package 'ocp-agent-ce' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=#!/bin/bash ... find_package 'ocp-agent-ce', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.756 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.executor.executor.SshExecutor : package name not found from host:ocp-agent-ce 2025-04-08 21:34:26.759 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.ocp.executor.executor.SshExecutor : uninstall skipped. No package found. rpmName=ocp-agent-ce 2025-04-08 21:34:26.768 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:26.870 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.910 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61 2025-04-08 21:34:26.941 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.953 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'rm -rf '"'"'/home/admin/ocp_agent'"'"'/*', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:26.960 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] .o.o.s.t.b.h.UninstallLegacyOcpAgentTask : [UninstallLegacyOcpAgentTask] legacy ocp agent uninstallation succeeded 2025-04-08 21:34:26.992 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000151, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:26.993 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,3d37b34bc834d8dd] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000156] 2025-04-08 21:34:27.271 INFO 26087 --- [prometheus-minute-schedule-1,0bdb1dd03927f6e7,ba1b2553f77ad935] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:27.745 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,f0a3d68a9afdf796] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000156, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:27.763 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,054fb27554472580] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:27.811 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,054fb27554472580] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000156, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:27.744+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000156}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:27.853 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,054fb27554472580] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000156, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:27.855 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,054fb27554472580] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000161] 2025-04-08 21:34:27.869 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,6177325e61f53450,245e824e41c305b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=6177325e61f53450, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:27.878 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-9,6177325e61f53450,245e824e41c305b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=6177325e61f53450, duration=12 ms] 2025-04-08 21:34:28.594 INFO 26087 --- [ocp-schedules-2,eb0c048ba6a7259a,3dd6b479bc919da0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:34:28.595 INFO 26087 --- [ocp-schedules-2,eb0c048ba6a7259a,3dd6b479bc919da0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:34:28.602 WARN 26087 --- [ocp-async-3,1100e82904ec0e27,19c807616119333c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:28.602 INFO 26087 --- [ocp-async-9,0682beb0934357ed,9f17a2c5af41891e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:28.760 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,f17af338548e2a4b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000161, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:28.773 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:28.794 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000161, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:28.758+08:00, sub_task_instance_name=Create random agent credential, sub_task_instance_id=1000161}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:28.799 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] c.o.o.s.c.CredentialServiceImpl : Begin to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=de0c9329-a439-4446-8245-f7c54c9a5b1a, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000006, authType=OCP-HMACSHA1)) 2025-04-08 21:34:28.807 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] c.o.o.s.c.CredentialServiceImpl : Success to create or replace credential:CreateCredentialParam(targetType=AGENT, validate=false, name=de0c9329-a439-4446-8245-f7c54c9a5b1a, description=agent auth credential, obJdbcCredentialProperty=null, sshCredentialProperty=null, agentCredentialProperty=AgentCredentialProperty(username=ocp_agent, hostId=1000006, authType=OCP-HMACSHA1)) 2025-04-08 21:34:28.811 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] c.o.o.s.t.b.h.CreateAgentCredentialTask : create agent credential success, credential id:1000013 2025-04-08 21:34:28.816 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000161, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:28.817 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,6c42abe059e5623f] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000154] 2025-04-08 21:34:29.058 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,d102b9d8e84697dc,dc85b50cba7fa50a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=d102b9d8e84697dc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:29.068 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-1,d102b9d8e84697dc,dc85b50cba7fa50a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=d102b9d8e84697dc, duration=13 ms] 2025-04-08 21:34:29.483 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:29.483 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:29.768 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,63a0d3c199f47309] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000154, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:29.821 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,5a668c6eb6d105c5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:29.872 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,5a668c6eb6d105c5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000154, context=Context{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:29.767+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=1000154}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:29.894 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,5a668c6eb6d105c5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000154, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:34:29.895 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,5a668c6eb6d105c5] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set rolling back subtask to ready, subtaskIds=[1000147] 2025-04-08 21:34:30.267 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,9b4e0bbdab3fa4e4,657086a00c4590b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=9b4e0bbdab3fa4e4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:30.271 INFO 26087 --- [prometheus-minute-schedule-1,891c1b95fec80fa2,fa1d48c3a51a5525] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:30.277 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,9b4e0bbdab3fa4e4,657086a00c4590b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=9b4e0bbdab3fa4e4, duration=13 ms] 2025-04-08 21:34:30.602 WARN 26087 --- [ocp-async-2,8c2b6227fd250563,1b549cb6078695a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:30.602 INFO 26087 --- [ocp-async-8,8d32b3cd100edc40,ce2b759a9034b5fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:30.806 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,e6a991ade2b61e14] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000147, operation:EXECUTE, state: RUNNING 2025-04-08 21:34:30.832 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:34:30.845 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000147, context=Context{parallelIdx=0, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, target_server_status=RUNNING, subtask_splitter=host_ids, ssh_port=22, service_name=ocpdb:1744114127, target_zone_status=RUNNING, task_instance_id=1000016, ob_connect_address=192.168.1.61:2881, task_operation=execute, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct, target_cluster_status=RUNNING, latest_execution_start_time=2025-04-08T21:34:30.805+08:00, sub_task_instance_name=Install ocp agent, sub_task_instance_id=1000147}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, executor=192.168.1.61 2025-04-08 21:34:30.847 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.s.t.b.host.InstallOcpAgentTask : [InstallOcpAgentTask] started 2025-04-08 21:34:30.850 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:34:30.852 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:34:30.854 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:34:30.856 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:34:30.873 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.service.task.util.AgentTaskUtils : Create ssh executor from task context 2025-04-08 21:34:30.895 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.service.task.util.AgentTaskUtils : CredentialId in context, use specific credential 2025-04-08 21:34:30.906 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:30.923 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:30.964 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:31.010 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.018 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 21:34:31.067 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.072 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62888$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.108 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:31.128 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.137 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61 2025-04-08 21:34:31.194 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.197 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo bash -c 'out=$(netstat -ant) && awk '"'"'$4~":62889$" { print $4"\t"$6 }'"'"' <<<"$out"', out=, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.205 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.s.c.AgentInstallationTaskService : download and install ocp-agent start... 2025-04-08 21:34:31.234 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: echo 1 on 192.168.1.61 2025-04-08 21:34:31.251 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: echo 1 on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=echo 1, out=1, err=, extOut=null, exitStatus=0) 2025-04-08 21:34:31.254 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand begin: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61 2025-04-08 21:34:31.285 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] com.oceanbase.ocp.common.ssh.SshUtils : SSH executeCommand end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61, result: SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:34:31.292 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.e.internal.template.SshTemplate : SSH execute end: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' on 192.168.1.61,result:SshResult(host=192.168.1.61, username=root, command=sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', out=, err=curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused , extOut=null, exitStatus=7) 2025-04-08 21:34:31.295 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.ocp.executor.executor.SshExecutor : execute ssh command failed, command:sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', return code:7, output:, err:curl: (7) Failed to connect to 192.168.1.61 port 80: Connection refused 2025-04-08 21:34:31.299 ERROR 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.ocp.executor.executor.SshExecutor : failed to execute ssh command, errMsg:[SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7, cause:null 2025-04-08 21:34:31.302 ERROR 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:34:31.317 WARN 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.executor.exception.SshCommandFailedException: [SshClient]: failed to execute ssh command: sudo curl -s -S -k -o '/tmp'/'ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm' 'http://192.168.1.61/api/v2/object-storage/sys-package/ocp-agent-ce-4.3.5-20250319105844.el7.x86_64.rpm', exit code: 7 at com.oceanbase.ocp.executor.executor.SshExecutor.execute(SshExecutor.java:429) at com.oceanbase.ocp.executor.executor.SshExecutor.lambda$downloadFile$2(SshExecutor.java:413) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:82) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:61) at com.oceanbase.ocp.common.lang.pattern.Retry.executeUntilWithLimit(Retry.java:56) at com.oceanbase.ocp.executor.executor.SshExecutor.downloadFile(SshExecutor.java:412) at com.oceanbase.ocp.executor.executor.SshExecutor.installPackage(SshExecutor.java:240) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService.installOcpAgentRpm(AgentInstallationTaskService.java:220) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$FastClassBySpringCGLIB$$f7a6037f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.compute.AgentInstallationTaskService$$EnhancerBySpringCGLIB$$c1a5a562.installOcpAgentRpm() at com.oceanbase.ocp.service.task.business.host.InstallOcpAgentTask.run(InstallOcpAgentTask.java:63) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:207) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:201) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:34:31.330 INFO 26087 --- [manual-subtask-executor16,29e4ae1f6d16bfd8,0db896e84cbeb346] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000147, operation:EXECUTE, state: FAILED 2025-04-08 21:34:31.465 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,ae2722f467ea541a,90ff18115a32f608] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=ae2722f467ea541a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:31.475 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-4,ae2722f467ea541a,90ff18115a32f608] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=ae2722f467ea541a, duration=13 ms] 2025-04-08 21:34:31.596 INFO 26087 --- [ocp-schedules-11,ce76dce54d6f248f,c2cbf73244e4ba17] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:34:32.276 INFO 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=1000016, name='Take over primary OB cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=1000003, tenantId=null, hostId=null, creator='admin', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={cluster_version=4.2.1.11, cluster_name=ocpdb, ssh_port=22, prohibit_rollback=false, service_name=ocpdb:1744114127, ob_connect_address=192.168.1.61:2881, cluster_type=PRIMARY, service_version=4.2.1.11, host_credential_id=1, ob_cluster_id=1744114127, cluster_id=1000003, root_sys_password=xxx service_type=OB_CLUSTER, obproxy_proxyro_password=xxx ob_data_dir=/oceanbasedata/ocpdb, connection_mode=direct}, listMap={add_region_ids=[1000004], server_ids=[1000006], add_idc_ids=[1000003], all_host_ids=[1000006], add_host_ids=[1000006], host_ids=[1000006], zone_names=[zone1]}}, createTime=2025-04-08T21:34:15+08:00, startTime=2025-04-08T21:34:15.498+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=1000143, name=Update ob cluster start time task, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.UpdateObStartTimeTask, seriesId=8, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000136, name=Update zone status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.zone.UpdateObZoneStatusTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000155, name=Wait cluster accessible, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.cluster.WaitClusterAccessibleTask, seriesId=3, startTime=null, endTime=null}, SubtaskInstanceEntity{id=1000151, name=Uninstall legacy ocp agent, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UninstallLegacyOcpAgentTask, seriesId=5, startTime=2025-04-08T21:34:23.731+08:00, endTime=2025-04-08T21:34:26.966+08:00}, SubtaskInstanceEntity{id=1000144, name=Wait node, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode 2025-04-08 21:34:32.296 WARN 26087 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Execute hook failed, ex: java.lang.NumberFormatException: null at java.lang.Long.parseLong(Long.java:552) at java.lang.Long.parseLong(Long.java:631) at com.oceanbase.ocp.core.task.common.Context.getLong(Context.java:196) at com.oceanbase.ocp.service.operation.OperationAlarmService.infoAfterTaskTerminate(OperationAlarmService.java:187) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lambda$executePostTaskHooks$5(RunningTaskWorker.java:161) at java.util.ArrayList.forEach(ArrayList.java:1259) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.executePostTaskHooks(RunningTaskWorker.java:159) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.lockAndScheduleRunningTask(RunningTaskWorker.java:113) at com.oceanbase.ocp.core.task.engine.coordinator.worker.RunningTaskWorker.work(RunningTaskWorker.java:79) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.nonExceptionRun(TaskCoordinator.java:118) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$initRunningTaskWorkers$3(TaskCoordinator.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:34:32.296 INFO 26087 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000016, status:FAILED 2025-04-08 21:34:32.483 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:32.484 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:32.602 WARN 26087 --- [ocp-async-1,4283e3cee1f0c329,975b203c0db5d0ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:32.602 INFO 26087 --- [ocp-async-4,aa3134fa300d810e,4fa6ec4891a7bca0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:32.660 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,a97d244f75bf5c65,ac92edc2886d23a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016?taskInstanceId=1000016, client=10.11.12.102, traceId=a97d244f75bf5c65, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=1000016,] 2025-04-08 21:34:32.670 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,a97d244f75bf5c65,ac92edc2886d23a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016 client=10.11.12.102, traceId=a97d244f75bf5c65, duration=14 ms] 2025-04-08 21:34:33.271 INFO 26087 --- [prometheus-minute-schedule-1,28cb468702561167,a74e7d09f893bcaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:34.602 WARN 26087 --- [ocp-async-10,ade07b28c13c1017,ed8c37257018a76b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:34.602 INFO 26087 --- [ocp-async-3,8a807837fef5802c,e3f5ef0d756ab608] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:34.955 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,de4ee6eed7d6a3f1,628f921428b173bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/1000016/subtasks/1000147/logSegments?taskInstanceId=1000016&subtaskInstanceId=1000147, client=10.11.12.102, traceId=de4ee6eed7d6a3f1, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=1000016,1000147,] 2025-04-08 21:34:34.957 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,de4ee6eed7d6a3f1,628f921428b173bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/1000016/subtasks/1000147/logSegments client=10.11.12.102, traceId=de4ee6eed7d6a3f1, duration=6 ms] 2025-04-08 21:34:35.484 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:35.485 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:36.271 INFO 26087 --- [prometheus-minute-schedule-1,cf9a770f72eb5efc,b519c42c49187ff1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:36.484 INFO 26087 --- [ocp-async-3,e660ca5831bbc5c1,e3dbf0a0f4122e1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:34:36.602 WARN 26087 --- [ocp-async-7,94b9b1356528614f,c624c31cc24402e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:36.602 INFO 26087 --- [ocp-async-2,b1c5e4a361d4ed2d,5d193271ae6a2f3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:34:36.622846+08:00, lastProcessTime=2025-04-08T21:34:26.622976+08:00 2025-04-08 21:34:36.697 INFO 26087 --- [ocp-schedules-14,1f93d2d9e34eb0b7,6368ab16afe38f92] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:34:36.761 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:34:36.622846+08:00 2025-04-08 21:34:37.742 INFO 26087 --- [ocp-schedules-26,aa77bb3eb08ca36d,06194a3665463d5d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:34:37.775 INFO 26087 --- [ocp-schedules-26,aa77bb3eb08ca36d,06194a3665463d5d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:34:37.775 INFO 26087 --- [ocp-schedules-26,aa77bb3eb08ca36d,06194a3665463d5d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:34:38.485 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:38.485 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:38.599 INFO 26087 --- [ocp-schedules-19,3d9dabe7c8c6dae3,22a855e4b815e6e7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:34:38.601 INFO 26087 --- [ocp-schedules-19,3d9dabe7c8c6dae3,22a855e4b815e6e7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:34:38.603 WARN 26087 --- [ocp-async-1,9f6dd9476610f3b5,797696fec1121030] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:38.603 INFO 26087 --- [ocp-async-4,b2255fd2430a8b24,573f04c8f4a25424] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:39.271 INFO 26087 --- [prometheus-minute-schedule-1,cd6e895591a69781,0da37cc0c2b497da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:40.001 INFO 26087 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:34:40.002 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:34:40.603 WARN 26087 --- [ocp-async-10,bdfd8eaddbaba2cf,78724f53821263e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:40.603 INFO 26087 --- [ocp-async-3,d0f3080ef0ba297c,faea3261d4b5675b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:41.485 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:41.486 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:41.619 INFO 26087 --- [ocp-schedules-13,92d5e9dfba1aefa4,3e55405049ffdd63] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:34:42.271 INFO 26087 --- [prometheus-minute-schedule-1,cf6b5bf13517de19,616f060756952cab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:42.603 WARN 26087 --- [ocp-async-8,fac6d9c705bf2851,902746c6d121ebf3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:42.603 INFO 26087 --- [ocp-async-9,330093cbf200e264,d0ee23fa52ede20a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:44.486 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:44.487 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:44.603 WARN 26087 --- [ocp-async-1,ab1a7a0d86bbaf22,86bd1f222bf82739] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:44.603 INFO 26087 --- [ocp-async-5,35828a49a9321360,381abec96ec100f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:45.271 INFO 26087 --- [prometheus-minute-schedule-1,dea8437726dd7d74,bbe7cce4adebc174] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:46.437 INFO 26087 --- [ocp-schedules-25,2e4f896423bff582,6e8a4999b2670f90] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:34:46.438 INFO 26087 --- [ocp-schedules-30,8c67f91650b7e7d9,0c4ee0c7fe7cef38] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:34:46.438 INFO 26087 --- [ocp-schedules-30,8c67f91650b7e7d9,0c4ee0c7fe7cef38] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:34:46.440 INFO 26087 --- [ocp-schedules-10,fe026d2d8b7b09f9,bbe6fbe890a4561b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:34:46.440 INFO 26087 --- [ocp-schedules-26,e07c8c3f9b0ff3f5,d38f2595431cb0f4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:34:46.440 INFO 26087 --- [ocp-schedules-10,fe026d2d8b7b09f9,bbe6fbe890a4561b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:34:46.440 INFO 26087 --- [ocp-schedules-26,e07c8c3f9b0ff3f5,d38f2595431cb0f4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:34:46.441 INFO 26087 --- [ocp-schedules-9,9a7ff54cb305c875,88f993375f01dee1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:34:46.441 INFO 26087 --- [ocp-schedules-27,ec7c568275793809,3ae2605f739b6747] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:34:46.441 INFO 26087 --- [ocp-schedules-12,f47c75832a4d794b,50858f2ab0aec5b4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:34:46.441 INFO 26087 --- [ocp-schedules-27,ec7c568275793809,3ae2605f739b6747] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:46.441 INFO 26087 --- [ocp-schedules-12,f47c75832a4d794b,50858f2ab0aec5b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:34:46.447 INFO 26087 --- [ocp-schedules-15,ba543ae3d03caf1d,ff83566cff948e12] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:34:46.447 INFO 26087 --- [ocp-schedules-15,80c0d46df9f5733e,2b54184284765ef3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:34:46.447 INFO 26087 --- [ocp-schedules-15,80c0d46df9f5733e,2b54184284765ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:46.452 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:34:46.452 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:34:46.484 INFO 26087 --- [ocp-async-2,8713387b06481635,6561a58c780a6d5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:34:46.564 INFO 26087 --- [ocp-schedules-23,21d49a794b119919,a450f481ff28cb65] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:34:46.603 WARN 26087 --- [ocp-async-10,a6017006653e6904,0854be5b51f40fd1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:46.603 INFO 26087 --- [ocp-async-6,59f2ca38fa69134f,dc405f83afe65e8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:34:46.623005+08:00, lastProcessTime=2025-04-08T21:34:36.622846+08:00 2025-04-08 21:34:46.698 INFO 26087 --- [ocp-schedules-17,199bfe327ba41aaa,86e323fe942625b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:34:46.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:34:46.623005+08:00 2025-04-08 21:34:47.487 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:47.487 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:48.271 INFO 26087 --- [prometheus-minute-schedule-1,a53a47717d11a491,0a43c4fb01cedae9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:48.603 INFO 26087 --- [ocp-async-8,610822f149de0c6b,fd6bcb5cd1c8a7b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:48.603 WARN 26087 --- [ocp-async-7,aa2369cd90141349,c9487029441d0489] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:48.606 INFO 26087 --- [ocp-schedules-19,1a750e2559efc007,451b2bfdb668277c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:34:48.607 INFO 26087 --- [ocp-schedules-19,1a750e2559efc007,451b2bfdb668277c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:34:50.487 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:50.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:50.603 WARN 26087 --- [ocp-async-1,7ffb152c22e61c43,5ae4391054d2fd86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:50.603 INFO 26087 --- [ocp-async-8,b24057cd48c2eba2,be590aebdc9d1460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:51.271 INFO 26087 --- [prometheus-minute-schedule-1,f2c8ab2480ed5c55,02c272e0ddecc0be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:51.644 INFO 26087 --- [ocp-schedules-10,b0323077d72a5e29,b543b9b726fd2bb1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:34:52.604 WARN 26087 --- [ocp-async-8,b3f8d775a62dfaa1,b406fb2e58336169] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:52.604 INFO 26087 --- [ocp-async-4,6276bbc18a66e9c2,0847c5414deb8b89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:53.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:53.488 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:54.271 INFO 26087 --- [prometheus-minute-schedule-1,a1d4c1361b6a3bbf,43c0709e509fac63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:54.604 WARN 26087 --- [ocp-async-6,377491c89fe996bf,51ee4d4ac087e4f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:54.604 INFO 26087 --- [ocp-async-4,b7ce67ac6aea8874,a1234844c3a3d255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:56.442 INFO 26087 --- [ocp-schedules-27,ec7c568275793809,3ae2605f739b6747] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:34:56.442 INFO 26087 --- [ocp-schedules-12,f47c75832a4d794b,50858f2ab0aec5b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:34:56.484 INFO 26087 --- [ocp-async-7,3ee4b4b9f2436c61,97037b3d6adf4ea6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:34:56.489 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:56.489 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:34:56.498 INFO 26087 --- [ocp-schedules-4,d7dd638eaa8a564a,7cd6f7b2f485afb8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:34:56.498 INFO 26087 --- [ocp-schedules-4,d7dd638eaa8a564a,7cd6f7b2f485afb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:34:56.499 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:34:56.604 INFO 26087 --- [ocp-async-1,2d1d92bee78b1c52,dfb6fdc660f29689] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:56.604 WARN 26087 --- [ocp-async-2,5b520fc50ce3072c,0c523474301d2e45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:34:56.623282+08:00, lastProcessTime=2025-04-08T21:34:46.623005+08:00 2025-04-08 21:34:56.699 INFO 26087 --- [ocp-schedules-18,27026c242b87fdbe,49cba1104e7573f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:34:56.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:34:56.623282+08:00 2025-04-08 21:34:57.271 INFO 26087 --- [prometheus-minute-schedule-1,286a743705be7a2b,dca6d1594f808545] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119300, interval=60 2025-04-08 21:34:57.776 INFO 26087 --- [ocp-schedules-5,13c2df3b85033ce0,51d3ab3db96bc999] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:34:57.781 INFO 26087 --- [ocp-schedules-5,13c2df3b85033ce0,51d3ab3db96bc999] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:34:57.781 INFO 26087 --- [ocp-schedules-5,13c2df3b85033ce0,51d3ab3db96bc999] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:34:58.604 WARN 26087 --- [ocp-async-1,e1940efd716c92c2,f6d0fde00761ebdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:34:58.604 INFO 26087 --- [ocp-async-3,9eeccb2f451f3e6c,90443b14edb5c0d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:34:58.615 INFO 26087 --- [ocp-schedules-9,e9612e1bee15408f,4f304c4de9dfe244] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:34:58.616 INFO 26087 --- [ocp-schedules-9,e9612e1bee15408f,4f304c4de9dfe244] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:34:59.489 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:34:59.490 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:00.266 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:30:01.0, nextRun=2025-04-08 21:35:00.0, currentDate=Tue Apr 08 21:35:00 CST 2025 2025-04-08 21:35:00.266 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:30:01.0, nextRunTime=2025-04-08 21:35:00.0} 2025-04-08 21:35:00.269 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:35:00.269 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:35:00.269+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:35:00.271 INFO 26087 --- [prometheus-minute-schedule-1,bde65d58776b161a,45daeee5847e5cb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:00.277 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:30:01.0, nextRun=2025-04-08 21:35:00.0, currentDate=Tue Apr 08 21:35:00 CST 2025 2025-04-08 21:35:00.277 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:30:01.0, nextRunTime=2025-04-08 21:35:00.0} 2025-04-08 21:35:00.282 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:35:00.283 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:35:00.283 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,8a1799066d72386c] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:35:00.604 INFO 26087 --- [ocp-async-3,5ce8e462e6334202,7f056177b095ac29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:00.604 WARN 26087 --- [ocp-async-7,bb4cb9e960d3651e,5e57854993c159e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:00.856 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,f0ebdc10a87d1ab8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000164, operation:EXECUTE, state: RUNNING 2025-04-08 21:35:00.881 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:35:00.884 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000164, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000017, task_operation=execute, latest_execution_start_time=2025-04-08T21:35:00.855+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000164}, listMap={}}, executor=192.168.1.61 2025-04-08 21:35:00.888 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.890 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 21:35:00.890 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.890 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.891 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.891 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.891 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.892 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.892 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:35:00.898 INFO 26087 --- [subtask-executor13,8ec1a44aeefd3a62,28d66012c4f88b99] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000164, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:35:01.666 INFO 26087 --- [ocp-schedules-18,ad64da4dbf91553e,bf5093d594f5785e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:35:01.790 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000017, status:SUCCESS 2025-04-08 21:35:01.791 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000017, status:SUCCESS 2025-04-08 21:35:02.490 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:02.490 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:02.604 WARN 26087 --- [ocp-async-5,660079500c435cea,815fde301de7ff6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:02.604 INFO 26087 --- [ocp-async-3,1532cafae8f5f71f,e7483b5130aaa96c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:03.271 INFO 26087 --- [prometheus-minute-schedule-1,7ab179a2e28e1629,b991bd37c81b17f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:04.604 WARN 26087 --- [ocp-async-10,ab37c44a0f94e80a,5fc45c64c86e2ac1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:04.604 INFO 26087 --- [ocp-async-1,5513591804b072ab,d54cd2f89228656b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:05.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:05.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:06.271 INFO 26087 --- [prometheus-minute-schedule-1,3b0ba359cf9716f4,74c0b6d329e51ca7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:06.272 INFO 26087 --- [prometheus-cache-monitor1,f051f71e778d27c8,4d35c3da870aa86e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1512,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:35:06.272 INFO 26087 --- [prometheus-cache-monitor1,f051f71e778d27c8,4d35c3da870aa86e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:35:06.484 INFO 26087 --- [ocp-async-1,0e9d43ed37dc4486,f9d284ca69298a0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:35:06.499 INFO 26087 --- [ocp-schedules-4,d7dd638eaa8a564a,7cd6f7b2f485afb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:35:06.605 WARN 26087 --- [ocp-async-9,9bc79e7a85df9159,1a104d7b639ef579] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:06.605 INFO 26087 --- [ocp-async-4,f492c9150dc6aad9,518517b494039387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:35:06.622483+08:00, lastProcessTime=2025-04-08T21:34:56.623282+08:00 2025-04-08 21:35:06.700 INFO 26087 --- [ocp-schedules-11,ace81a38380c7c6b,672239e54d68098b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:35:06.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:35:06.622483+08:00 2025-04-08 21:35:08.491 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:08.492 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:08.605 WARN 26087 --- [ocp-async-8,626f51af05e22b9c,57a0093b8966879d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:08.605 INFO 26087 --- [ocp-async-4,fcb95c5ee7dc69a7,94f04760ee4af74b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:08.625 INFO 26087 --- [ocp-schedules-18,76c2e57d43cb7896,0b375ef8cf98c66d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:35:08.626 INFO 26087 --- [ocp-schedules-18,76c2e57d43cb7896,0b375ef8cf98c66d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:35:09.271 INFO 26087 --- [prometheus-minute-schedule-1,576fce905f0d74fc,0037fcbe7b61a08c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:10.001 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:35:10.002 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:35:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:35:10.605 WARN 26087 --- [ocp-async-1,960df64cf3872a76,d50a31616fbb94e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:10.605 INFO 26087 --- [ocp-async-3,502008ee9b5d3649,5b9f4435b52e7843] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:11.492 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:11.493 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:11.690 INFO 26087 --- [ocp-schedules-9,5251df7b05041932,86bc6440bd2176a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:35:12.271 INFO 26087 --- [prometheus-minute-schedule-1,8faaabd2d60debbc,2d6af6b7c2843443] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:12.419 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=30, affectRows=30 2025-04-08 21:35:12.419 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=29, validCount=29 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=39, affectRows=39 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=51, affectRows=51 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=40, affectRows=40 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=43, affectRows=43 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=40, validCount=40 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=39, validCount=39 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=42, validCount=42 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=51, validCount=51 2025-04-08 21:35:12.420 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=57, affectRows=57 2025-04-08 21:35:12.421 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=59, affectRows=59 2025-04-08 21:35:12.421 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119240, nodeCount=61, affectRows=61 2025-04-08 21:35:12.421 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=57, validCount=57 2025-04-08 21:35:12.421 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=59, validCount=59 2025-04-08 21:35:12.421 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119240, nodeCount=60, validCount=60 2025-04-08 21:35:12.605 WARN 26087 --- [ocp-async-9,cdd56a2fdf1ffbd1,8bcc9d21551cc90a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:12.605 INFO 26087 --- [ocp-async-6,ffd1966a6bd7f86e,5eb70349fdb46f8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:14.493 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:14.494 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:14.605 WARN 26087 --- [ocp-async-8,82554e380e9b17a4,7d560bdf1a7a2f07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:14.605 INFO 26087 --- [ocp-async-2,3e79ce57023c6abb,4cecdf1916b3a8d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:15.271 INFO 26087 --- [prometheus-minute-schedule-1,dc4c1c21ed400114,b5ce922150838868] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:16.433 INFO 26087 --- [ocp-schedules-9,a29aaeb6d134371e,0b1e71e01a909efd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:35:16.434 INFO 26087 --- [ocp-schedules-30,0bad22214c567051,3f36ea36358c2abd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:35:16.436 INFO 26087 --- [ocp-schedules-24,e50cd133449c36af,ad99ea44fd70509b] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:35:16.436+08:00, tenantSize=0 2025-04-08 21:35:16.437 INFO 26087 --- [ocp-schedules-8,10ac17ce5218b5e6,94dcbd66091b7ede] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:35:16.437 INFO 26087 --- [ocp-schedules-5,be12ca852badf94d,187cb4d90a95bb82] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:35:16.437 INFO 26087 --- [ocp-schedules-8,10ac17ce5218b5e6,94dcbd66091b7ede] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:35:16.439 INFO 26087 --- [ocp-schedules-20,4c9d7e661b51a8b2,9218943b39595388] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:35:16.439 INFO 26087 --- [ocp-schedules-21,b90873db7276329c,90f2d097008d8449] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:35:16.439 INFO 26087 --- [ocp-schedules-20,4c9d7e661b51a8b2,9218943b39595388] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:35:16.439 INFO 26087 --- [ocp-schedules-21,b90873db7276329c,90f2d097008d8449] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:35:16.440 INFO 26087 --- [ocp-schedules-30,0bad22214c567051,3f36ea36358c2abd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:35:16.441 INFO 26087 --- [ocp-schedules-4,dad64c9181a8c19d,45f60be46cca98d6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:35:16.441 INFO 26087 --- [ocp-schedules-28,73561efb30d686f6,e4e271fa7650beab] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:35:16.441 INFO 26087 --- [ocp-schedules-4,dad64c9181a8c19d,45f60be46cca98d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:16.441 INFO 26087 --- [ocp-schedules-28,73561efb30d686f6,e4e271fa7650beab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:35:16.442 INFO 26087 --- [ocp-schedules-9,a29aaeb6d134371e,0b1e71e01a909efd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:35:16.453 INFO 26087 --- [ocp-schedules-15,80c0d46df9f5733e,2b54184284765ef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:35:16.453 INFO 26087 --- [ocp-schedules-15,44b779fa97e58194,3826fb93b1c95d88] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:35:16.454 INFO 26087 --- [ocp-schedules-15,44b779fa97e58194,3826fb93b1c95d88] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:16.467 INFO 26087 --- [ocp-schedules-26,2711e8dbdb150536,7a3d973060dce99f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:35:16.471 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:35:16.471 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:35:16.480 INFO 26087 --- [ocp-schedules-27,4b3ac13e592f26cc,e6e1408f3c37c749] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 21:35:16.480 INFO 26087 --- [ocp-schedules-27,4b3ac13e592f26cc,e6e1408f3c37c749] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 21:35:16.484 INFO 26087 --- [ocp-async-1,1dc37bd1b2b74ed5,99366eb4ee1a693c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:35:16.504 INFO 26087 --- [ocp-schedules-16,c4428aab20df9041,88328f04e607d220] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1514,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:35:16.504 INFO 26087 --- [ocp-schedules-16,c4428aab20df9041,88328f04e607d220] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:35:16.529 INFO 26087 --- [ocp-schedules-29,1414c7c88c61db0c,c6bc05a0d4bc1c12] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:35:16.556 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:35:16.556 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:35:16.561 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:35:16.561 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:35:16.561 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:35:16.563 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:35:16.563 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:35:16.563 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:35:16.565 INFO 26087 --- [ocp-schedules-5,6084b32745c05ce1,1514797f82597024] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:35:16.566 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:35:16.566 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:35:16.566 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:35:16.568 INFO 26087 --- [ocp-schedules-8,69a7d344745705d0,634fb470d6f43790] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:35:16.572 INFO 26087 --- [ocp-schedules-20,7ae7a4f25a45adb2,1496ea62af274680] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:35:16.572 INFO 26087 --- [ocp-schedules-20,7ae7a4f25a45adb2,1496ea62af274680] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:35:16.582 INFO 26087 --- [ocp-schedules-30,293476947deb2726,a68094e025cc0e37] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:35:16.582 INFO 26087 --- [ocp-schedules-30,293476947deb2726,a68094e025cc0e37] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:35:16.587 INFO 26087 --- [ocp-schedules-21,8a3cc9fd76ccaae7,15a69ba211154f64] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:35:16.587+08:00, clusters=[1000003] 2025-04-08 21:35:16.587 INFO 26087 --- [ocp-schedules-21,8a3cc9fd76ccaae7,15a69ba211154f64] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:35:16.588 INFO 26087 --- [ocp-schedules-1,26d3544c890960f2,8fbc2d645f1b4ba2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:35:16.589 INFO 26087 --- [ocp-schedules-12,4819ff8c5f8bac4a,56aee84455b58a75] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:35:16.589+08:00, clusters=[1000003] 2025-04-08 21:35:16.590 INFO 26087 --- [ocp-schedules-12,4819ff8c5f8bac4a,56aee84455b58a75] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:35:16.605 WARN 26087 --- [ocp-async-3,a5bcdde23e0d6ef2,6d801fbb802a4cc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:16.605 INFO 26087 --- [ocp-async-9,36e4b428053c9d04,a77bd0cd5f36d1e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:16.608 INFO 26087 --- [ocp-schedules-17,f34428368d1a41e0,cce96e63b344fc7a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:35:16.608 INFO 26087 --- [ocp-schedules-17,f34428368d1a41e0,cce96e63b344fc7a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:35:16.609 INFO 26087 --- [ocp-schedules-2,4f2ab272bc7cc6ba,74ba8a44806b6d69] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:35:16.610 INFO 26087 --- [ocp-schedules-2,4f2ab272bc7cc6ba,74ba8a44806b6d69] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:35:16.610 INFO 26087 --- [ocp-schedules-2,4f2ab272bc7cc6ba,74ba8a44806b6d69] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:35:16.613 INFO 26087 --- [ocp-schedules-17,f34428368d1a41e0,cce96e63b344fc7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:35:16.613 INFO 26087 --- [ocp-schedules-17,f34428368d1a41e0,cce96e63b344fc7a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:35:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:35:16.622574+08:00, lastProcessTime=2025-04-08T21:35:06.622483+08:00 2025-04-08 21:35:16.701 INFO 26087 --- [ocp-schedules-23,110bf87c52f377db,425ddab080c41a98] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:35:16.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:35:16.622574+08:00 2025-04-08 21:35:16.754 INFO 26087 --- [ocp-schedules-11,dcd5d10fc67ed2ad,866d9ec4f32354ca] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:35:16.755 INFO 26087 --- [ocp-schedules-11,dcd5d10fc67ed2ad,866d9ec4f32354ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:16.766 ERROR 26087 --- [ocp-schedules-11,dcd5d10fc67ed2ad,866d9ec4f32354ca] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:35:16.766 INFO 26087 --- [ocp-schedules-11,dcd5d10fc67ed2ad,866d9ec4f32354ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:35:16.776 INFO 26087 --- [ocp-schedules-6,23c391e7bce8da93,c9898daf28b67935] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:35:16.786 INFO 26087 --- [ocp-schedules-7,596848d5d442d990,44d45cbcc8ec4fe8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:35:16.787 INFO 26087 --- [ocp-schedules-7,596848d5d442d990,44d45cbcc8ec4fe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:16.803 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:35:16.805 ERROR 26087 --- [ocp-schedules-7,596848d5d442d990,44d45cbcc8ec4fe8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:35:16.805 INFO 26087 --- [ocp-schedules-7,596848d5d442d990,44d45cbcc8ec4fe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:35:16.813 INFO 26087 --- [ocp-schedules-14,3a9d7d4d1ce7b780,f7e937a532d51a0f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:35:16.813 INFO 26087 --- [ocp-schedules-14,3a9d7d4d1ce7b780,f7e937a532d51a0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:35:16.814 INFO 26087 --- [ocp-schedules-14,3a9d7d4d1ce7b780,f7e937a532d51a0f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:35:16.814 INFO 26087 --- [ocp-schedules-14,3a9d7d4d1ce7b780,f7e937a532d51a0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:35:16.814 INFO 26087 --- [ocp-schedules-14,3a9d7d4d1ce7b780,f7e937a532d51a0f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:35:16.814 INFO 26087 --- [ocp-schedules-14,3a9d7d4d1ce7b780,f7e937a532d51a0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:35:16.900 INFO 26087 --- [ocp-schedules-19,2bdb58756a987ec5,a16387bca63bd790] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:35:17.377 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:35:17.377 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:35:17.377 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.378 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:35:17.378 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.379 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.379 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.379 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:35:17.391 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.391 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.392 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.392 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:35:17.401 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.402 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.402 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.402 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:35:17.413 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.413 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.413 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.414 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:35:17.424 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.425 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.425 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.425 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:35:17.430 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.431 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.431 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.431 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:35:17.435 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.435 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.435 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.435 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:35:17.439 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.439 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.439 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.440 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:35:17.444 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.445 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.445 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.445 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:35:17.449 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.449 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.450 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.450 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:35:17.454 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.454 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.455 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.455 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:35:17.462 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.462 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.462 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.463 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:35:17.474 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.474 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.474 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.475 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:35:17.479 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.479 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.479 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.479 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:35:17.484 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.484 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.485 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.485 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:35:17.490 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.490 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.490 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.491 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:35:17.494 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:17.494 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:17.494 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.494 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.494 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.495 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:35:17.498 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.498 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.498 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.498 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:35:17.502 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.502 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.502 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.502 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:35:17.506 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.506 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.506 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.507 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:35:17.510 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.510 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.510 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.510 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:35:17.514 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.514 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.514 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.514 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:35:17.517 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.517 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.517 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.518 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:35:17.521 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.521 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.521 INFO 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:35:17.521 INFO 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:35:17.521 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.522 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:35:17.524 INFO 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:35:17.525 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.525 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.525 WARN 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:35:17.526 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.526 INFO 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:35:17.526 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:35:17.527 WARN 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:35:17.529 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.529 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.529 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.530 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:35:17.533 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.533 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.534 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.534 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:35:17.535 INFO 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.e.i.connector.ConnectorHolder : [ConnectorHolder]: connector put to cache, key=ConnectorKey(protocol=http, hostAddress=192.168.1.61, httpPort=62889, sshPort=null, authentication=Authentication(httpAuth=HttpAuthentication(authType=OCP-HMACSHA1, basicAuthConfig=null, digestAuthConfig=DigestAuthConfig(username=ocp_agent), formBaseAuthConfig=null, sslAuthConfig=null), sshAuth=null), proxy=null), estimated size:12 2025-04-08 21:35:17.535 INFO 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:35:17.537 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.537 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.538 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.538 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:35:17.541 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.541 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.542 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.542 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:35:17.545 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.545 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.546 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.546 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:35:17.549 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.549 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.549 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.550 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:35:17.553 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.554 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.555 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.555 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:35:17.559 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.559 WARN 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:35:17.559 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.559 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.559 ERROR 26087 --- [ocp-schedules-12,7c59ebcc850a437b,e1b47e3390850769] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:35:17.559 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:35:17.563 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.563 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.563 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.564 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:35:17.573 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.573 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.574 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.574 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:35:17.581 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.581 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.581 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.582 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:35:17.586 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.586 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.586 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.587 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:35:17.591 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.591 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.591 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.592 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:35:17.594 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.594 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.595 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.595 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:35:17.597 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.598 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.598 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.598 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:35:17.601 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.601 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.602 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.602 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:35:17.604 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.605 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.605 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.605 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:35:17.612 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.612 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.613 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.613 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:35:17.630 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.630 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.631 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.631 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:35:17.636 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.636 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.636 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.637 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:35:17.642 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.642 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.642 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.643 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:35:17.647 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.647 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.648 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.648 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:35:17.651 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.651 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.651 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.651 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:35:17.654 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.655 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.655 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.655 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:35:17.658 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:35:17.658 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:35:17.659 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.659 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:35:17.660 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.660 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.660 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.660 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:35:17.664 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.664 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:35:17.664 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:35:17.664 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:35:17.667 INFO 26087 --- [ocp-schedules-30,6a30ceeaf87a5246,ed401da384f96834] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:35:17.782 INFO 26087 --- [ocp-schedules-22,1d3d5d60a9d75d1d,676780d651e7ca8b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:35:17.787 INFO 26087 --- [ocp-schedules-22,1d3d5d60a9d75d1d,676780d651e7ca8b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:35:17.787 INFO 26087 --- [ocp-schedules-22,1d3d5d60a9d75d1d,676780d651e7ca8b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:35:18.271 INFO 26087 --- [prometheus-minute-schedule-1,9750bee0d331d880,a7a4e2595945b314] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:18.605 WARN 26087 --- [ocp-async-2,83cfc9560dff71a4,cec669afb89095d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:18.605 INFO 26087 --- [ocp-async-6,255aacf11368f46f,77b8688e16b1fff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:18.656 INFO 26087 --- [ocp-schedules-8,086b5a01d8670b72,d682ee821b3b902a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:35:18.657 INFO 26087 --- [ocp-schedules-8,086b5a01d8670b72,d682ee821b3b902a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:35:20.494 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:20.495 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:35:20.606 WARN 26087 --- [ocp-async-1,383d753b39b42cc1,3fd50d1fd0fd17cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:20.606 INFO 26087 --- [ocp-async-4,62caacef7f5d53e5,70215e63b5611d7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:21.271 INFO 26087 --- [prometheus-minute-schedule-1,d86ebaf9d03322fa,da2d828b9bd1f4db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:21.714 INFO 26087 --- [ocp-schedules-20,784c55796246135c,26ac2b75c6bc88b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:35:22.606 WARN 26087 --- [ocp-async-3,f7a0fbaa2c16b48b,46d6c7bfbfcd4c2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:22.606 INFO 26087 --- [ocp-async-7,aef8045ad24ff254,52549d2dfd747c6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:23.495 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:23.496 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:24.271 INFO 26087 --- [prometheus-minute-schedule-1,fa4e8f4141ac76ea,3fb855f090e74055] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:24.606 WARN 26087 --- [ocp-async-2,606bddaea9b24b7a,a2ee184a3b11261c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:24.606 INFO 26087 --- [ocp-async-8,a08a55c617074e72,402d43b8c5deb37b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:26.442 INFO 26087 --- [ocp-schedules-28,73561efb30d686f6,e4e271fa7650beab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:35:26.442 INFO 26087 --- [ocp-schedules-4,dad64c9181a8c19d,45f60be46cca98d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:35:26.484 INFO 26087 --- [ocp-async-8,d70cf1d1a5b324c5,ee60fa6105c9c650] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:35:26.496 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:26.496 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:26.606 WARN 26087 --- [ocp-async-6,a8ea563b30f2460d,c24e765ad20c74ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:26.606 INFO 26087 --- [ocp-async-4,a66ca24a287e48f5,27ec535141375667] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:35:26.622982+08:00, lastProcessTime=2025-04-08T21:35:16.622574+08:00 2025-04-08 21:35:26.701 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:35:26.622982+08:00 2025-04-08 21:35:26.702 INFO 26087 --- [ocp-schedules-29,243768971e25804b,3cbf83652ae43417] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:35:27.271 INFO 26087 --- [prometheus-minute-schedule-1,ca2eee3d3453fb61,5dbfe20ee58c9132] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:28.606 WARN 26087 --- [ocp-async-10,9172142a0c255e2b,22df62da4a8aa614] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:28.606 INFO 26087 --- [ocp-async-2,bde819d3991468ca,b3a107888227efe3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:28.683 INFO 26087 --- [ocp-schedules-17,af013214dac34d90,14cbf1be1f415686] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:35:28.684 INFO 26087 --- [ocp-schedules-17,af013214dac34d90,14cbf1be1f415686] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:35:29.496 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:29.497 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:30.271 INFO 26087 --- [prometheus-minute-schedule-1,a15aa8c7b75bf6ec,cb30a1291887f4e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:30.606 WARN 26087 --- [ocp-async-8,e72bec1dd96d0c10,53bb571bcecd963c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:30.606 INFO 26087 --- [ocp-async-9,4541348c1963bf83,69ceeaa3bcd56e33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:31.746 INFO 26087 --- [ocp-schedules-20,58951d9949227cfc,e254e7a5e0d5e96c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:35:32.497 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:32.498 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:32.606 WARN 26087 --- [ocp-async-9,9ca701413b72420d,8a57f23670f1a7e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:32.606 INFO 26087 --- [ocp-async-3,b5dea6ee1ef96586,9c723bac2f878a24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:33.271 INFO 26087 --- [prometheus-minute-schedule-1,a52759259f41e5e3,d92dbc42f60137b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:34.607 WARN 26087 --- [ocp-async-10,7da35acf4fe3a66b,a97ffac2e5ff9eff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:34.607 INFO 26087 --- [ocp-async-4,93d12df976e561a9,e4cd7d3f3a0e3ce9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:34.894 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:35:34.899 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:35:34.899 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 21:35:35.498 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:35.498 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:36.271 INFO 26087 --- [prometheus-minute-schedule-1,7a2a76ac44a36041,d8aaa78ae5f6b2f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:36.484 INFO 26087 --- [ocp-async-4,695c4e00cb6bff3a,2d126e408960b618] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:35:36.607 INFO 26087 --- [ocp-async-8,80e5b23e648aa934,f3f7d90e08ce2845] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:36.607 WARN 26087 --- [ocp-async-5,54c3ee4cf363b0e5,ded42030a5779b0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:35:36.624162+08:00, lastProcessTime=2025-04-08T21:35:26.622982+08:00 2025-04-08 21:35:36.703 INFO 26087 --- [ocp-schedules-1,dbe3c4c5eb8fa067,9720128524eee14e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:35:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:35:36.624162+08:00 2025-04-08 21:35:37.788 INFO 26087 --- [ocp-schedules-3,d9642ec6dd300be6,f2c4cb98a6467a6c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:35:37.792 INFO 26087 --- [ocp-schedules-3,d9642ec6dd300be6,f2c4cb98a6467a6c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:35:37.792 INFO 26087 --- [ocp-schedules-3,d9642ec6dd300be6,f2c4cb98a6467a6c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:35:38.499 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:38.499 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:38.607 WARN 26087 --- [ocp-async-9,771ad8471ee61018,10f948d455c9dc52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:38.607 INFO 26087 --- [ocp-async-3,2139888775d7c639,18fd843f4f90e0a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:38.736 INFO 26087 --- [ocp-schedules-23,1ffc48904ea75b68,26a31155d5c56b2b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:35:38.737 INFO 26087 --- [ocp-schedules-23,1ffc48904ea75b68,26a31155d5c56b2b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:35:39.271 INFO 26087 --- [prometheus-minute-schedule-1,ed05a19f8252f304,485ed8486fea686a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:40.001 INFO 26087 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:35:40.002 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:35:40.607 WARN 26087 --- [ocp-async-4,dd9022c96d770fd7,db08892f69fbde83] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:40.607 INFO 26087 --- [ocp-async-7,ad7cb40a176bb47f,b24edc807645b633] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:41.499 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:41.500 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:41.766 INFO 26087 --- [ocp-schedules-8,4274ee8ea7f2e94f,225d63836e5c450a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:35:42.271 INFO 26087 --- [prometheus-minute-schedule-1,c596f156b39b60a6,434ff08234354e10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:42.607 WARN 26087 --- [ocp-async-5,f92722911efbf8cd,b4139cb7b8681d1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:42.607 INFO 26087 --- [ocp-async-2,b7e6110bcdc7bdbf,a29d5508a9d78d55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:44.500 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:44.501 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:44.607 WARN 26087 --- [ocp-async-6,5e37ae426ef89a5b,fefb1448aed10cf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:44.607 INFO 26087 --- [ocp-async-9,ca759f7c5e2a10b0,e2c6102cab25ce4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:45.271 INFO 26087 --- [prometheus-minute-schedule-1,b93225af796a6a31,98ea71980676981b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:46.437 INFO 26087 --- [ocp-schedules-12,508372762f777e67,7bcf2f15ae2c8e1c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:35:46.437 INFO 26087 --- [ocp-schedules-11,ff7f2f357f471661,10cea5240dce937b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:35:46.437 INFO 26087 --- [ocp-schedules-11,ff7f2f357f471661,10cea5240dce937b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:35:46.440 INFO 26087 --- [ocp-schedules-9,61ff4eaf7650f7b3,b11b86eef5dede25] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:35:46.440 INFO 26087 --- [ocp-schedules-9,61ff4eaf7650f7b3,b11b86eef5dede25] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:35:46.440 INFO 26087 --- [ocp-schedules-6,7bee1bf45f6c675b,48c9927ff94c669e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:35:46.440 INFO 26087 --- [ocp-schedules-6,7bee1bf45f6c675b,48c9927ff94c669e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:35:46.441 INFO 26087 --- [ocp-schedules-11,999016d5b11b2fa7,d7650caa57e74868] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:35:46.441 INFO 26087 --- [ocp-schedules-10,056a6732ae976131,861bdaaa17c3b70e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:35:46.441 INFO 26087 --- [ocp-schedules-11,999016d5b11b2fa7,d7650caa57e74868] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:46.441 INFO 26087 --- [ocp-schedules-10,056a6732ae976131,861bdaaa17c3b70e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:35:46.444 INFO 26087 --- [ocp-schedules-25,1e98bae70c17b087,788b5077e0e5607d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:35:46.472 INFO 26087 --- [ocp-schedules-15,44b779fa97e58194,3826fb93b1c95d88] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:35:46.472 INFO 26087 --- [ocp-schedules-15,c88adce4a802bb81,ab2d30f045c56cc1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:35:46.473 INFO 26087 --- [ocp-schedules-15,c88adce4a802bb81,ab2d30f045c56cc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:35:46.477 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:35:46.477 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:35:46.485 INFO 26087 --- [ocp-async-8,8d1f39a50a505d27,bc4194f60dc8151a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:35:46.567 INFO 26087 --- [ocp-schedules-2,52335aa6bb3ac47e,08e0fdb65a7691d8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:35:46.608 WARN 26087 --- [ocp-async-7,036355727fdb929a,8ddff58419e64daa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:46.608 INFO 26087 --- [ocp-async-4,9f031a2e4351be70,1e1863f4197c0e14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:35:46.623297+08:00, lastProcessTime=2025-04-08T21:35:36.624162+08:00 2025-04-08 21:35:46.705 INFO 26087 --- [ocp-schedules-7,ee0bc608ad6b23be,8c33084c2317aaf2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:35:46.732 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:35:46.623297+08:00 2025-04-08 21:35:47.501 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:47.501 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:48.271 INFO 26087 --- [prometheus-minute-schedule-1,931b1b9b958f01e6,40c956ed9e4fbe87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:48.608 WARN 26087 --- [ocp-async-1,2161d19f6b3f532c,95076e619d88dd7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:48.608 INFO 26087 --- [ocp-async-2,fd0c5abb86712260,a342f0bbdade9166] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:48.746 INFO 26087 --- [ocp-schedules-30,cd5a9d75f9e23c90,5901953274bccb46] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:35:48.747 INFO 26087 --- [ocp-schedules-30,cd5a9d75f9e23c90,5901953274bccb46] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:35:50.501 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:50.502 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:50.608 WARN 26087 --- [ocp-async-6,20eca1ba130b7acf,8c797692bdf16537] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:50.608 INFO 26087 --- [ocp-async-9,d3efb8c455f9973f,a6a6296d70d4f012] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:51.271 INFO 26087 --- [prometheus-minute-schedule-1,98208ed549df1921,35e49ec78c2c1af2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:51.796 INFO 26087 --- [ocp-schedules-29,2476143686781cc7,f74334154a45fdfa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:35:52.608 WARN 26087 --- [ocp-async-3,0057f54aebf1ab8a,39c09b1c09e3f129] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:52.608 INFO 26087 --- [ocp-async-7,92098f7ca6b14abc,0a2e95d710477d81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:53.502 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:53.503 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:53.624 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-6,5a6d53fa68fa8b0c,8707d2742e203f21] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=5a6d53fa68fa8b0c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744119353621625,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 21:35:53.626 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-6,5a6d53fa68fa8b0c,8707d2742e203f21] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=5a6d53fa68fa8b0c, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744119353621625,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=3 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 21:35:54.271 INFO 26087 --- [prometheus-minute-schedule-1,56a40fd4ba5de2fe,95c318705d0e4ee1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:54.608 WARN 26087 --- [ocp-async-4,d3c1411f55ad885f,bc9e62a738350776] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:54.608 INFO 26087 --- [ocp-async-1,c722f8fa18a461fb,a3a53e96d65b35e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:56.442 INFO 26087 --- [ocp-schedules-10,056a6732ae976131,861bdaaa17c3b70e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:35:56.442 INFO 26087 --- [ocp-schedules-11,999016d5b11b2fa7,d7650caa57e74868] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:35:56.485 INFO 26087 --- [ocp-async-7,0dea5d357761a2a3,81781ae9e066cdf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:35:56.503 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:56.503 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:35:56.608 WARN 26087 --- [ocp-async-6,a4245555e3ef64a0,59d0ad9259fcc5c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:56.608 INFO 26087 --- [ocp-async-8,343e2718670ceb12,15f5129bff3e2b55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:35:56.622288+08:00, lastProcessTime=2025-04-08T21:35:46.623297+08:00 2025-04-08 21:35:56.706 INFO 26087 --- [ocp-schedules-25,45b6a5a09b434b9a,64711c34c178096b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:35:56.746 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:35:56.622288+08:00 2025-04-08 21:35:57.271 INFO 26087 --- [prometheus-minute-schedule-1,9b29690f3fc24a39,d5c54697b9645af1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119360, interval=60 2025-04-08 21:35:57.793 INFO 26087 --- [ocp-schedules-22,da65395882432053,132419e347021948] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:35:57.844 INFO 26087 --- [ocp-schedules-22,da65395882432053,132419e347021948] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:35:57.844 INFO 26087 --- [ocp-schedules-22,da65395882432053,132419e347021948] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:35:58.608 WARN 26087 --- [ocp-async-10,f7b6692450343801,491f9b0680c08806] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:35:58.608 INFO 26087 --- [ocp-async-8,6339e02bdbd33376,43616c2391e35635] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:35:58.750 INFO 26087 --- [ocp-schedules-23,0d802fe6ffd1e068,bd89415c2dd75257] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:35:58.751 INFO 26087 --- [ocp-schedules-23,0d802fe6ffd1e068,bd89415c2dd75257] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:35:59.503 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:35:59.504 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:00.271 INFO 26087 --- [prometheus-minute-schedule-1,cce7d2463df9791b,5eb760565ece0181] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:00.608 WARN 26087 --- [ocp-async-7,b4457a72a8b05427,d644840ed7426dfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:00.608 INFO 26087 --- [ocp-async-5,9417f4458c8bfdda,36c2f55fd07a8a96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:01.816 INFO 26087 --- [ocp-schedules-11,d2d14b3bd9997444,92c86cd76abbc395] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:36:02.504 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:02.505 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:02.609 WARN 26087 --- [ocp-async-6,bb245c4f88b568f8,ef61b7ecc3aecde9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:02.609 INFO 26087 --- [ocp-async-5,4aa675967bcfb774,2007803f21ecb02a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:03.271 INFO 26087 --- [prometheus-minute-schedule-1,729d0602287b55b5,24a3c83d8d3a91f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:04.609 WARN 26087 --- [ocp-async-2,c391a602db37e343,13357808b1e6771c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:04.609 INFO 26087 --- [ocp-async-4,e259258190bf87a1,9c291361f2e711dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:05.505 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:05.506 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:06.271 INFO 26087 --- [prometheus-minute-schedule-1,0d986a17a2c59f62,e22aed918dae6ed8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:06.272 INFO 26087 --- [prometheus-cache-monitor1,2ec89d4d70a73ad0,583c87a4cb949d55] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1514,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:36:06.272 INFO 26087 --- [prometheus-cache-monitor1,2ec89d4d70a73ad0,583c87a4cb949d55] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:36:06.485 INFO 26087 --- [ocp-async-4,66940499cc88ab48,e14add0f2fe8c491] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:36:06.499 INFO 26087 --- [ocp-schedules-18,30f4e0ccdb2fc46e,16d853c9196e1673] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:36:06.500 INFO 26087 --- [ocp-schedules-18,30f4e0ccdb2fc46e,16d853c9196e1673] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:06.500 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:36:06.604 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:32:06.578, end=2025-04-08T21:34:06.593} 2025-04-08 21:36:06.606 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:36:06.606 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:36:06.609 WARN 26087 --- [ocp-async-7,43df2c7dcc5f2d8a,696c32d78a0bc7ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:06.609 INFO 26087 --- [ocp-async-1,7e6459feeeacf666,6928cae653a8005c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:36:06.624007+08:00, lastProcessTime=2025-04-08T21:35:56.622288+08:00 2025-04-08 21:36:06.707 INFO 26087 --- [ocp-schedules-5,73cf3d44efced82a,f8f39450e07361e4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:36:06.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:36:06.624007+08:00 2025-04-08 21:36:08.506 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:08.506 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:08.609 WARN 26087 --- [ocp-async-1,07a449ed4ca1d421,2011edefd9be1c3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:08.609 INFO 26087 --- [ocp-async-9,8ec125a3390b56fd,b7770f91e88da686] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:08.760 INFO 26087 --- [ocp-schedules-4,0c633d83a930ec9c,ae2b1901de9b1f77] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:36:08.761 INFO 26087 --- [ocp-schedules-4,0c633d83a930ec9c,ae2b1901de9b1f77] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:36:09.271 INFO 26087 --- [prometheus-minute-schedule-1,5c50886a9e5c08a6,87ec4cd3830ffa3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:10.001 INFO 26087 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:36:10.002 INFO 26087 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:36:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:36:10.609 WARN 26087 --- [ocp-async-2,cd34d8a6c8d40370,7bb947475daa3ff9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:10.609 INFO 26087 --- [ocp-async-10,4ab665d2d63ab70b,c8e151978a5ecd9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:11.507 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:11.507 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:11.839 INFO 26087 --- [ocp-schedules-29,8cbb5d5f94741aee,7a996e58746f8889] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:36:12.271 INFO 26087 --- [prometheus-minute-schedule-1,5350a5680652c5fc,ca81c3ce2c2b1280] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:12.454 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119300, nodeCount=100, affectRows=100 2025-04-08 21:36:12.454 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119300, nodeCount=97, affectRows=97 2025-04-08 21:36:12.454 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119300, nodeCount=96, affectRows=96 2025-04-08 21:36:12.454 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119300, nodeCount=87, affectRows=87 2025-04-08 21:36:12.455 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119300, nodeCount=99, validCount=99 2025-04-08 21:36:12.455 INFO 26087 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119300, nodeCount=87, validCount=87 2025-04-08 21:36:12.455 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119300, nodeCount=95, validCount=95 2025-04-08 21:36:12.455 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119300, nodeCount=96, validCount=96 2025-04-08 21:36:12.609 WARN 26087 --- [ocp-async-3,4c8e4d3fb096d54d,c837ad2a7eee84b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:12.609 INFO 26087 --- [ocp-async-7,7d15d7b8ed9fe335,baf1f064498001d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:14.507 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:14.508 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:14.609 WARN 26087 --- [ocp-async-1,109dd51d728f3d4a,d48003160c6ea295] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:14.609 INFO 26087 --- [ocp-async-7,4c358b39ce677b3b,3d99af692a96bba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:15.271 INFO 26087 --- [prometheus-minute-schedule-1,60ac757be842884b,2328fe996b58954c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:16.433 INFO 26087 --- [ocp-schedules-27,62ad083b57958c4c,0961660b54415d06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:36:16.434 INFO 26087 --- [ocp-schedules-16,3178fe1e7f8831fd,b01e0cba5b0030a4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:36:16.437 INFO 26087 --- [ocp-schedules-22,6a49476fb3d4f1e2,95cf5a5c0d89405d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:36:16.437 INFO 26087 --- [ocp-schedules-22,6a49476fb3d4f1e2,95cf5a5c0d89405d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:36:16.437 INFO 26087 --- [ocp-schedules-22,364c0855bd492e36,52cf073428d9469e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:36:16.439 INFO 26087 --- [ocp-schedules-30,f6ab49fa123dba93,2c623613c1a300ac] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:36:16.439 INFO 26087 --- [ocp-schedules-30,f6ab49fa123dba93,2c623613c1a300ac] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:36:16.439 INFO 26087 --- [ocp-schedules-26,b653ffb56070d1c4,d77e8a5d2a39341a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:36:16.439 INFO 26087 --- [ocp-schedules-26,b653ffb56070d1c4,d77e8a5d2a39341a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:36:16.440 INFO 26087 --- [ocp-schedules-16,3178fe1e7f8831fd,b01e0cba5b0030a4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:36:16.441 INFO 26087 --- [ocp-schedules-1,a510f0954b7e768d,94255779804b4bda] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:36:16.441 INFO 26087 --- [ocp-schedules-8,b148a81ac4cd74c7,f152e3e10c5cbad6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:36:16.441 INFO 26087 --- [ocp-schedules-1,a510f0954b7e768d,94255779804b4bda] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:16.441 INFO 26087 --- [ocp-schedules-8,b148a81ac4cd74c7,f152e3e10c5cbad6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:36:16.478 INFO 26087 --- [ocp-schedules-15,c88adce4a802bb81,ab2d30f045c56cc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:36:16.478 INFO 26087 --- [ocp-schedules-15,e71e004d56ee1c2b,519932784cd1d7f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:36:16.479 INFO 26087 --- [ocp-schedules-15,e71e004d56ee1c2b,519932784cd1d7f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:16.483 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:36:16.483 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:36:16.485 INFO 26087 --- [ocp-async-2,a53ebca6b1ba9807,47bde1c75620d6e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:36:16.500 INFO 26087 --- [ocp-schedules-18,30f4e0ccdb2fc46e,16d853c9196e1673] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:36:16.512 INFO 26087 --- [ocp-schedules-27,62ad083b57958c4c,0961660b54415d06] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:36:16.526 INFO 26087 --- [ocp-schedules-4,384db3a6f4c8e5a7,aa56546891da7afc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:36:16.530 INFO 26087 --- [ocp-schedules-11,3972f01641d49f58,d0134bac10cda2b5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:36:16.568 INFO 26087 --- [ocp-schedules-10,96a5ce27d9f02c2b,8f7a8380191c6c50] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:36:16.574 INFO 26087 --- [ocp-schedules-5,7a5be918d732120a,c7fc793a8c300307] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:36:16.574 INFO 26087 --- [ocp-schedules-5,7a5be918d732120a,c7fc793a8c300307] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:36:16.584 INFO 26087 --- [ocp-schedules-22,e7dba4fb38b8a539,d14f8599c219716a] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:36:16.584 INFO 26087 --- [ocp-schedules-22,e7dba4fb38b8a539,d14f8599c219716a] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:36:16.592 INFO 26087 --- [ocp-schedules-30,7742221f7d4e1384,1b10e85a13e340f0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:36:16.592+08:00, clusters=[1000003] 2025-04-08 21:36:16.592 INFO 26087 --- [ocp-schedules-30,7742221f7d4e1384,1b10e85a13e340f0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:36:16.593 INFO 26087 --- [ocp-schedules-26,e6668f8f2c998185,92a26a3ed414506f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:36:16.595 INFO 26087 --- [ocp-schedules-16,746139bb999bb975,778b01d263896941] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:36:16.595+08:00, clusters=[1000003] 2025-04-08 21:36:16.597 INFO 26087 --- [ocp-schedules-16,746139bb999bb975,778b01d263896941] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:36:16.610 WARN 26087 --- [ocp-async-8,66ce7ceefc05255d,924f2102c07c485f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:16.610 INFO 26087 --- [ocp-async-10,91ac769d70e1fc66,292442a50bce7642] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:16.610 INFO 26087 --- [ocp-schedules-9,f99b6db4e815b430,e15f9c61ee9cedb0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:36:16.612 INFO 26087 --- [ocp-schedules-9,f99b6db4e815b430,e15f9c61ee9cedb0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:36:16.612 INFO 26087 --- [ocp-schedules-9,f99b6db4e815b430,e15f9c61ee9cedb0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:36:16.613 INFO 26087 --- [ocp-schedules-24,6bca70c811e927a4,70624c2a1f9e45e9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:36:16.613 INFO 26087 --- [ocp-schedules-24,6bca70c811e927a4,70624c2a1f9e45e9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:36:16.618 INFO 26087 --- [ocp-schedules-24,6bca70c811e927a4,70624c2a1f9e45e9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:36:16.618 INFO 26087 --- [ocp-schedules-24,6bca70c811e927a4,70624c2a1f9e45e9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:36:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:36:16.622554+08:00, lastProcessTime=2025-04-08T21:36:06.624007+08:00 2025-04-08 21:36:16.707 INFO 26087 --- [ocp-schedules-7,858ef1854e528dc4,09c9d187207caf1c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:36:16.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:36:16.622554+08:00 2025-04-08 21:36:16.776 INFO 26087 --- [ocp-schedules-25,5e34f269cb54b2af,db0af0a2019693cd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:36:16.777 INFO 26087 --- [ocp-schedules-25,5e34f269cb54b2af,db0af0a2019693cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:16.786 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:36:16.786 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:36:16.805 INFO 26087 --- [ocp-schedules-3,2e123fced999074b,42a68531b1d207b6] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:36:16.806 INFO 26087 --- [ocp-schedules-3,2e123fced999074b,42a68531b1d207b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:16.814 INFO 26087 --- [ocp-schedules-2,842fff83947e65b0,793c305047ba96d8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:36:16.814 INFO 26087 --- [ocp-schedules-2,842fff83947e65b0,793c305047ba96d8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:36:16.815 INFO 26087 --- [ocp-schedules-2,842fff83947e65b0,793c305047ba96d8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:36:16.815 INFO 26087 --- [ocp-schedules-2,842fff83947e65b0,793c305047ba96d8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:36:16.815 INFO 26087 --- [ocp-schedules-2,842fff83947e65b0,793c305047ba96d8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:36:16.815 INFO 26087 --- [ocp-schedules-2,842fff83947e65b0,793c305047ba96d8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:36:16.821 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:36:16.823 ERROR 26087 --- [ocp-schedules-3,2e123fced999074b,42a68531b1d207b6] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:36:16.823 INFO 26087 --- [ocp-schedules-3,2e123fced999074b,42a68531b1d207b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:36:16.906 INFO 26087 --- [ocp-schedules-6,7d8d538c7b00d8ff,52e4a07615b6e754] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:36:17.508 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:17.508 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:17.563 INFO 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:36:17.563 INFO 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:36:17.565 INFO 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:36:17.567 WARN 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:36:17.567 INFO 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:36:17.568 WARN 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:36:17.571 INFO 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:36:17.572 WARN 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:36:17.573 ERROR 26087 --- [ocp-schedules-5,beb62e3193180202,71ef6c3a5454e96b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:36:17.845 INFO 26087 --- [ocp-schedules-24,f93a1aca39f80a40,34bd2efac24aab46] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:36:17.863 INFO 26087 --- [ocp-schedules-24,f93a1aca39f80a40,34bd2efac24aab46] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:36:17.863 INFO 26087 --- [ocp-schedules-24,f93a1aca39f80a40,34bd2efac24aab46] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:36:18.271 INFO 26087 --- [prometheus-minute-schedule-1,ab44a94f60738b2b,719e22e9ad3e929c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:18.610 INFO 26087 --- [ocp-async-6,9fab655065c00432,c5ca705944f83a20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:18.610 WARN 26087 --- [ocp-async-7,e9a308c50d18ccd6,58fcc79e008ff4ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:18.818 INFO 26087 --- [ocp-schedules-12,0cb1ef111fcca9e2,f971a0b8667667a6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:36:18.820 INFO 26087 --- [ocp-schedules-12,0cb1ef111fcca9e2,f971a0b8667667a6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:36:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:36:20.508 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:20.509 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:20.610 WARN 26087 --- [ocp-async-4,6a9ff43b101b4bf9,6f8434379607d7a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:20.610 INFO 26087 --- [ocp-async-5,e9b2dc2ac38b7992,c32fc9d3ad2ed33e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:21.271 INFO 26087 --- [prometheus-minute-schedule-1,0ffcfaca9724a90b,e052ba8cf9fddf48] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:21.858 INFO 26087 --- [ocp-schedules-11,686409585605f93c,eddbbceebfbd9f37] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:36:22.610 WARN 26087 --- [ocp-async-3,eeb77bc93ff0a8c7,b16a384ac8cf179a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:22.610 INFO 26087 --- [ocp-async-2,d05b357868f367ce,a38c6386999c9887] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:23.509 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:23.510 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:24.271 INFO 26087 --- [prometheus-minute-schedule-1,d7038ae8d104c3d6,494348c6615683d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:24.610 WARN 26087 --- [ocp-async-7,7fd4d2ee88d501fb,8444b3f14d073811] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:24.610 INFO 26087 --- [ocp-async-1,db8a06b7ab7a15ec,c1988cfc9aa0582d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:26.442 INFO 26087 --- [ocp-schedules-8,b148a81ac4cd74c7,f152e3e10c5cbad6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:36:26.442 INFO 26087 --- [ocp-schedules-1,a510f0954b7e768d,94255779804b4bda] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:36:26.485 INFO 26087 --- [ocp-async-1,db09b218887e898d,8f1e4c7770c3c153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:36:26.510 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:26.510 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:26.610 WARN 26087 --- [ocp-async-5,67bfe59ffa3a01bf,ce0e9ce82293c9b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:26.610 INFO 26087 --- [ocp-async-3,bb95bbe3d18530fa,754e26d0543fd9fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:36:26.622132+08:00, lastProcessTime=2025-04-08T21:36:16.622554+08:00 2025-04-08 21:36:26.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:36:26.622132+08:00 2025-04-08 21:36:26.708 INFO 26087 --- [ocp-schedules-8,049a742c9d6f2ddf,c81bd65c575869ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:36:27.271 INFO 26087 --- [prometheus-minute-schedule-1,17341c9b59ea38f3,ee61fa3101d85fa4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:28.611 WARN 26087 --- [ocp-async-2,a0efae1621ca470f,41dc31e36cd57871] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:28.611 INFO 26087 --- [ocp-async-3,031f2e72d44a1245,360adf54ce2e056c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:28.865 INFO 26087 --- [ocp-schedules-24,b204f87c76f4b9d2,b7edbbf1adf1c677] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:36:28.867 INFO 26087 --- [ocp-schedules-24,b204f87c76f4b9d2,b7edbbf1adf1c677] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:36:29.510 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:29.511 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:29.733 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:36:29.733 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:36:29.736 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:36:29.736 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:36:29.736 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:36:29.736 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:36:29.740 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:36:29.740 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:36:29.740 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:36:29.741 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:36:29.762 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:36:29.767 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:36:29.767 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:36:29.767 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:36:29.767 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:36:29.767 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:36:29.772 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:36:30.079 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:36:30.271 INFO 26087 --- [prometheus-minute-schedule-1,c70b797c24a6dced,264520fd275eea89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:30.405 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.664557271S 2025-04-08 21:36:30.405 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:36:30.461 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:36:30.461 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:36:30.485 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:36:30.524 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:36:30.524 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:36:30.524 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:36:30.524 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:36:30.524 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:36:30.524 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:36:30.532 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:36:30.611 WARN 26087 --- [ocp-async-3,dc9614a509c55869,35243af26a1ad725] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:30.611 INFO 26087 --- [ocp-async-10,d306f3fa6e668a8e,1bcde04a48db8c03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:30.626 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] o.s.boot.SpringApplication : Started application in 0.89 seconds (JVM running for 2096.083) 2025-04-08 21:36:30.627 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.890191488S 2025-04-08 21:36:30.893 INFO 26087 --- [ocp-schedules-29,211e29617197f6df,683e8f608e487b51] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:36:31.886 INFO 26087 --- [ocp-schedules-10,9837781c90ffe7c0,0af553b0a441dffd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:36:32.511 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:32.511 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:32.611 WARN 26087 --- [ocp-async-5,887b68bff533a9c5,c53c07c63cadd86d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:32.611 INFO 26087 --- [ocp-async-4,c590c8357cacd7c4,a467ca1978e212c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:33.271 INFO 26087 --- [prometheus-minute-schedule-1,3e6cca986655b10a,acbb647fa41caae0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:34.611 WARN 26087 --- [ocp-async-7,1a986301c9d55ab8,a7a94f74e1578abe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:34.611 INFO 26087 --- [ocp-async-2,6b719005a1a6df36,4e556e02f6857b7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:35.511 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:35.512 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:36.271 INFO 26087 --- [prometheus-minute-schedule-1,f5de32a62e899afd,bb97dc2245abee69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:36.485 INFO 26087 --- [ocp-async-9,ce283bd6dff54b77,b46a36c1e4e96b60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:36:36.611 WARN 26087 --- [ocp-async-1,7d7f3ea5e239f697,5327c6ac8fdc57ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:36.611 INFO 26087 --- [ocp-async-3,8fca73c3d1683a5a,b4494c7cf0bb0ef7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:36:36.622646+08:00, lastProcessTime=2025-04-08T21:36:26.622132+08:00 2025-04-08 21:36:36.709 INFO 26087 --- [ocp-schedules-24,6195bbe1eba14635,3f65b47a3c5a4a99] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:36:36.713 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:36:36.622646+08:00 2025-04-08 21:36:37.865 INFO 26087 --- [ocp-schedules-4,95c14159bfac301f,76dfdaaa714c35f0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:36:37.882 INFO 26087 --- [ocp-schedules-4,95c14159bfac301f,76dfdaaa714c35f0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:36:37.882 INFO 26087 --- [ocp-schedules-4,95c14159bfac301f,76dfdaaa714c35f0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:36:38.512 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:38.512 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:38.611 WARN 26087 --- [ocp-async-6,59d36eb8b9b6c5e9,fbb833be5e58c623] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:38.611 INFO 26087 --- [ocp-async-4,69ced51247096c12,ee87f6729a2c34a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:38.869 INFO 26087 --- [ocp-schedules-19,d46446dea8d6ba2d,ba8b3a616a081c2c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:36:38.870 INFO 26087 --- [ocp-schedules-19,d46446dea8d6ba2d,ba8b3a616a081c2c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:36:39.271 INFO 26087 --- [prometheus-minute-schedule-1,ace27e84587204ba,6d296530684c60ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:40.001 INFO 26087 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:36:40.001 INFO 26087 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:36:40.611 WARN 26087 --- [ocp-async-8,c0023abd4ca7972f,10401faf5a16874a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:40.611 INFO 26087 --- [ocp-async-7,76fb1f29431a8150,7420d7908380b283] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:41.513 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:41.513 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:41.909 INFO 26087 --- [ocp-schedules-21,285d7f54a5b2f0f4,a7027f73ae11ed23] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:36:42.271 INFO 26087 --- [prometheus-minute-schedule-1,b4b8725747013fb2,52aac56fb4989464] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:42.612 WARN 26087 --- [ocp-async-1,4c95d4f22e2d8232,686b8cda24c1a10e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:42.612 INFO 26087 --- [ocp-async-2,39902701c065c0a1,4bf67934ca079e58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:44.513 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:44.514 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:44.612 WARN 26087 --- [ocp-async-6,c48e29837474fce8,f37d1147fdfd879d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:44.612 INFO 26087 --- [ocp-async-3,4aab1ef5376cc699,a23d699d12fb273d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:45.271 INFO 26087 --- [prometheus-minute-schedule-1,bef782b978eff12b,478f71ee131bc897] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:46.437 INFO 26087 --- [ocp-schedules-6,3025275ffa38386a,4133f49d746de6ae] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:36:46.437 INFO 26087 --- [ocp-schedules-6,3025275ffa38386a,4133f49d746de6ae] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:36:46.437 INFO 26087 --- [ocp-schedules-6,0a7e0f4a020e4f4b,1d75e1633e5b6093] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:36:46.440 INFO 26087 --- [ocp-schedules-14,ac69e90bf71f2566,9cac904aa2f18f81] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:36:46.440 INFO 26087 --- [ocp-schedules-14,ac69e90bf71f2566,9cac904aa2f18f81] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:36:46.440 INFO 26087 --- [ocp-schedules-12,5068db5536e533a2,6f1227de7fc2a34d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:36:46.440 INFO 26087 --- [ocp-schedules-12,5068db5536e533a2,6f1227de7fc2a34d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:36:46.441 INFO 26087 --- [ocp-schedules-17,376cc5b6db9376ca,19f94b2eb685310e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:36:46.441 INFO 26087 --- [ocp-schedules-8,d9e244056edb9e38,512054fd2b23e932] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:36:46.441 INFO 26087 --- [ocp-schedules-11,641532cab80ca743,6b221bd8969ff9be] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:36:46.441 INFO 26087 --- [ocp-schedules-17,376cc5b6db9376ca,19f94b2eb685310e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:46.441 INFO 26087 --- [ocp-schedules-11,641532cab80ca743,6b221bd8969ff9be] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:36:46.483 INFO 26087 --- [ocp-schedules-15,e71e004d56ee1c2b,519932784cd1d7f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:36:46.484 INFO 26087 --- [ocp-schedules-15,9ca5f28f2f43f28f,8a2b566c344c4850] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:36:46.484 INFO 26087 --- [ocp-schedules-15,9ca5f28f2f43f28f,8a2b566c344c4850] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:36:46.485 INFO 26087 --- [ocp-async-5,6fbc3bfdeebbb9ca,d10603ff70eb2994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:36:46.490 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:36:46.490 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:36:46.569 INFO 26087 --- [ocp-schedules-24,b3fc8a371690e61f,fbb0debb016aa4b2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:36:46.612 WARN 26087 --- [ocp-async-8,f419b8272869a570,4a70825b8b131090] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:46.612 INFO 26087 --- [ocp-async-7,5403b8e9407bc1f4,981aaa0c75d50486] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:36:46.622943+08:00, lastProcessTime=2025-04-08T21:36:36.622646+08:00 2025-04-08 21:36:46.710 INFO 26087 --- [ocp-schedules-19,56bb602bbebcc2eb,4bd9da76644bb73e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:36:46.740 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:36:46.622943+08:00 2025-04-08 21:36:47.514 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:47.515 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:48.271 INFO 26087 --- [prometheus-minute-schedule-1,2c61b7f4e45a0148,b2478fc62bc4fbf5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:48.612 WARN 26087 --- [ocp-async-10,24ba25b588803f42,dd56b9a12bb74aa6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:48.612 INFO 26087 --- [ocp-async-2,f36842f4985337a5,acd1c47db2539106] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:48.888 INFO 26087 --- [ocp-schedules-18,db6cc473498f7cbc,70e536fedee3a28a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:36:48.890 INFO 26087 --- [ocp-schedules-18,db6cc473498f7cbc,70e536fedee3a28a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:36:50.515 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:50.515 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:50.612 WARN 26087 --- [ocp-async-6,ea59780c9064e22b,a09d7b8cbe8618f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:50.612 INFO 26087 --- [ocp-async-5,0b71862c879be19a,6c62a8ea9d0f20ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:51.271 INFO 26087 --- [prometheus-minute-schedule-1,8973da2b399dc65c,ad514f7d2bea84fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:51.938 INFO 26087 --- [ocp-schedules-19,b66697661cdc8a99,26016f3345a5611e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:36:52.612 WARN 26087 --- [ocp-async-5,86dc336ce0965b4b,ee6436cac726e0ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:52.612 INFO 26087 --- [ocp-async-5,f97ddbccc9a10cad,a3a03dae168644fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:53.516 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:53.516 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:54.271 INFO 26087 --- [prometheus-minute-schedule-1,af49aff80040a72e,6d31c137b96ee339] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:54.612 WARN 26087 --- [ocp-async-10,9d9fdb1ef36a3e46,5ce8252974c2ba8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:54.612 INFO 26087 --- [ocp-async-5,1053053453d419c5,8826426453792d99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:56.442 INFO 26087 --- [ocp-schedules-11,641532cab80ca743,6b221bd8969ff9be] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:36:56.442 INFO 26087 --- [ocp-schedules-17,376cc5b6db9376ca,19f94b2eb685310e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:36:56.486 INFO 26087 --- [ocp-async-7,8b082bafcf6fcd6a,cc153ffe0c84accb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:36:56.516 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:56.517 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:36:56.613 WARN 26087 --- [ocp-async-2,73c523263a09e497,130dfc6433eb617e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:56.613 INFO 26087 --- [ocp-async-2,7920e3a1c117716f,9e86ac8c24d7aed2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:36:56.623640+08:00, lastProcessTime=2025-04-08T21:36:46.622943+08:00 2025-04-08 21:36:56.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:36:56.623640+08:00 2025-04-08 21:36:56.712 INFO 26087 --- [ocp-schedules-13,e8edf2f82b8fa283,258fe57c9218e1d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:36:57.271 INFO 26087 --- [prometheus-minute-schedule-1,66d3eee50d35108d,070751ff96701bab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119420, interval=60 2025-04-08 21:36:57.883 INFO 26087 --- [ocp-schedules-28,253ca12f05851bba,31486130a17bfad9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:36:57.926 INFO 26087 --- [ocp-schedules-28,253ca12f05851bba,31486130a17bfad9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:36:57.926 INFO 26087 --- [ocp-schedules-28,253ca12f05851bba,31486130a17bfad9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:36:58.613 WARN 26087 --- [ocp-async-9,ee0a8043e3147578,f0921e2954b5da2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:36:58.613 INFO 26087 --- [ocp-async-8,8727646eacb50717,588a4aa5e9f951d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:36:58.894 INFO 26087 --- [ocp-schedules-12,45f40bd33bdc7b19,ec13d2f28fa292d8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:36:58.895 INFO 26087 --- [ocp-schedules-12,45f40bd33bdc7b19,ec13d2f28fa292d8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:36:59.517 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:36:59.517 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:00.271 INFO 26087 --- [prometheus-minute-schedule-1,04375eda1f0361b5,308d2bcbd17f89f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:00.613 WARN 26087 --- [ocp-async-7,a55f7fe34bd57985,cd3f366469c69418] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:00.613 INFO 26087 --- [ocp-async-1,53c15470480f4ac5,b60b9f350b18b808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:01.958 INFO 26087 --- [ocp-schedules-30,05130a1c6e12ca65,a6df6ffc7d3ca92d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:37:02.517 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:02.518 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:02.613 WARN 26087 --- [ocp-async-3,dcb8f621f339ec1b,be6b8f6c2245c459] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:02.613 INFO 26087 --- [ocp-async-1,10bc752fb84d025d,d9ea4b574967f817] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:03.271 INFO 26087 --- [prometheus-minute-schedule-1,6b1250decd96ba2a,ef4130b9270fbb24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:04.613 WARN 26087 --- [ocp-async-10,996077c609ab06c7,bfb6af16d68f0e06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:04.613 INFO 26087 --- [ocp-async-9,e4276a9b6d78631d,4c568b9305c39f20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:05.518 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:05.519 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:06.271 INFO 26087 --- [prometheus-minute-schedule-1,3fbedcc01a0dc32d,e7b300ac20ac15be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:06.272 INFO 26087 --- [prometheus-cache-monitor1,e005a6c8a2b5c93e,d26f998fb5a1bbfd] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1516,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:37:06.272 INFO 26087 --- [prometheus-cache-monitor1,e005a6c8a2b5c93e,d26f998fb5a1bbfd] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:37:06.486 INFO 26087 --- [ocp-async-2,1b0ae60ba00e1974,92ad189bdc3553d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:37:06.613 INFO 26087 --- [ocp-async-7,3fb5a8077b0cd255,96cdcb9cc321d6b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:06.613 WARN 26087 --- [ocp-async-8,a0ee3b7d5415100c,f257409762e66273] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:37:06.622729+08:00, lastProcessTime=2025-04-08T21:36:56.623640+08:00 2025-04-08 21:37:06.713 INFO 26087 --- [ocp-schedules-7,a12dfe329bb7fe49,d5a1728d368a512e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:37:06.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:37:06.622729+08:00 2025-04-08 21:37:08.519 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:08.519 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:08.613 WARN 26087 --- [ocp-async-6,83f59176f87cbc54,a7ea397123c488ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:08.614 INFO 26087 --- [ocp-async-3,0a367d5f1f3ecedb,dd5bac305288d2e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:08.914 INFO 26087 --- [ocp-schedules-12,a623324fc6283454,4bad52e8edcbe2fd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:37:08.915 INFO 26087 --- [ocp-schedules-12,a623324fc6283454,4bad52e8edcbe2fd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:37:09.271 INFO 26087 --- [prometheus-minute-schedule-1,c7548ec04bbc020d,28e83547276a6b05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:10.001 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:37:10.002 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:37:10.521 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:37:10.614 WARN 26087 --- [ocp-async-1,b7a7775a9a2226d4,281b663ae694af4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:10.614 INFO 26087 --- [ocp-async-10,a286340ba0698e86,4563bd98a6d886e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:11.519 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:11.520 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:11.982 INFO 26087 --- [ocp-schedules-16,a515d27ad7918492,0b3afac995e75c88] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:37:12.271 INFO 26087 --- [prometheus-minute-schedule-1,73df79ed9ef6d277,2355a8def85fdfd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:12.463 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119360, nodeCount=132, affectRows=132 2025-04-08 21:37:12.464 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119360, nodeCount=130, validCount=130 2025-04-08 21:37:12.478 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119360, nodeCount=101, affectRows=101 2025-04-08 21:37:12.478 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119360, nodeCount=147, affectRows=147 2025-04-08 21:37:12.479 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119360, nodeCount=101, validCount=101 2025-04-08 21:37:12.479 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119360, nodeCount=146, validCount=146 2025-04-08 21:37:12.614 WARN 26087 --- [ocp-async-8,58d8a774f41a907e,f8a3bdf693ff2a22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:12.614 INFO 26087 --- [ocp-async-5,a088064de54e6f23,de5cfec1f3e99ff2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:14.520 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:14.521 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:14.614 WARN 26087 --- [ocp-async-4,1e7b625511b39b60,65a900522e0e8711] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:14.614 INFO 26087 --- [ocp-async-5,c2a6836fa1b03fc5,693087ab517878cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:15.271 INFO 26087 --- [prometheus-minute-schedule-1,5bf1f7a637118b39,a1fc284e07c965c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:16.433 INFO 26087 --- [ocp-schedules-3,e21d403f2210e0f7,b1d6266865536d64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:37:16.434 INFO 26087 --- [ocp-schedules-18,cb98869d0a1de3c4,3878a69332ba11e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:37:16.437 INFO 26087 --- [ocp-schedules-13,eee299ca66ed9dea,afff85f6645452c3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:37:16.437 INFO 26087 --- [ocp-schedules-13,eee299ca66ed9dea,afff85f6645452c3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:37:16.437 INFO 26087 --- [ocp-schedules-8,82fce9570269f528,a8c82f15f4356984] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:37:16.439 INFO 26087 --- [ocp-schedules-10,7c671f3e78e4e2a9,c1f5687ffcde6c4f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:37:16.439 INFO 26087 --- [ocp-schedules-10,7c671f3e78e4e2a9,c1f5687ffcde6c4f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:37:16.439 INFO 26087 --- [ocp-schedules-4,f6f39329ca9e4f36,0b09af76eea9d4d9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:37:16.439 INFO 26087 --- [ocp-schedules-4,f6f39329ca9e4f36,0b09af76eea9d4d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:37:16.441 INFO 26087 --- [ocp-schedules-30,288f214177f437b4,b5b222cdb54d6b74] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:37:16.441 INFO 26087 --- [ocp-schedules-29,51652085a1f1cd42,9bc9a7515d71a570] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:37:16.441 INFO 26087 --- [ocp-schedules-18,cb98869d0a1de3c4,3878a69332ba11e9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:37:16.441 INFO 26087 --- [ocp-schedules-30,288f214177f437b4,b5b222cdb54d6b74] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:16.441 INFO 26087 --- [ocp-schedules-29,51652085a1f1cd42,9bc9a7515d71a570] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:37:16.470 INFO 26087 --- [ocp-schedules-21,aeddf5953e7a3df0,c860dc50acfb0fa6] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2025-04-08 21:37:16.470 INFO 26087 --- [ocp-schedules-21,aeddf5953e7a3df0,c860dc50acfb0fa6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-04-08 21:37:16.481 INFO 26087 --- [ocp-schedules-23,3cdd12ee13e76211,b3552f7deb703fd9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:37:16.482 INFO 26087 --- [ocp-schedules-23,3cdd12ee13e76211,b3552f7deb703fd9] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:37:16.483 INFO 26087 --- [ocp-schedules-3,e21d403f2210e0f7,b1d6266865536d64] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:37:16.486 INFO 26087 --- [ocp-async-1,c0614dc4b758f0e4,8cf15f7b7376d9e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:37:16.490 INFO 26087 --- [ocp-schedules-15,9ca5f28f2f43f28f,8a2b566c344c4850] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:37:16.490 INFO 26087 --- [ocp-schedules-15,2b3282e2c61c29b5,9a17f5602bd7f56d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:37:16.491 INFO 26087 --- [ocp-schedules-15,2b3282e2c61c29b5,9a17f5602bd7f56d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:16.495 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:37:16.495 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:37:16.500 INFO 26087 --- [ocp-schedules-17,97f86bfbfd782773,e15591161a926374] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:37:16.501 INFO 26087 --- [ocp-schedules-6,ef12da93ad3cf1f6,4050dea926995ac1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:37:16.501 INFO 26087 --- [ocp-schedules-6,ef12da93ad3cf1f6,4050dea926995ac1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:16.501 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:37:16.505 INFO 26087 --- [ocp-schedules-20,1c40471860358225,3d33f8420bf7fcfb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1518,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:37:16.505 INFO 26087 --- [ocp-schedules-20,1c40471860358225,3d33f8420bf7fcfb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:37:16.531 INFO 26087 --- [ocp-schedules-13,c35587bfd9810fb8,f8af145c0ac69578] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:37:16.570 INFO 26087 --- [ocp-schedules-10,56ab4c46a5f6aa53,a9d129ff79c5ed69] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:37:16.576 INFO 26087 --- [ocp-schedules-4,b5fef78eb8ef8f6c,67ced35b99f5ca5c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:37:16.576 INFO 26087 --- [ocp-schedules-4,b5fef78eb8ef8f6c,67ced35b99f5ca5c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:37:16.586 INFO 26087 --- [ocp-schedules-18,8c17f5fa3ecd0bb4,dfcaade327ff66d6] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:37:16.586 INFO 26087 --- [ocp-schedules-18,8c17f5fa3ecd0bb4,dfcaade327ff66d6] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:37:16.597 INFO 26087 --- [ocp-schedules-7,3fcfad87e015e35c,21709b62f9e9fd57] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:37:16.597+08:00, clusters=[1000003] 2025-04-08 21:37:16.597 INFO 26087 --- [ocp-schedules-7,3fcfad87e015e35c,21709b62f9e9fd57] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:37:16.598 INFO 26087 --- [ocp-schedules-26,ecd934f64a6bbc8c,09634f09b6ffe01e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:37:16.601 INFO 26087 --- [ocp-schedules-9,1be799f76a22fdd3,d1766c92a40ae726] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:37:16.601+08:00, clusters=[1000003] 2025-04-08 21:37:16.602 INFO 26087 --- [ocp-schedules-9,1be799f76a22fdd3,d1766c92a40ae726] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:37:16.612 INFO 26087 --- [ocp-schedules-21,76da350dabda3c10,b60ed2c741306163] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:37:16.613 INFO 26087 --- [ocp-schedules-21,76da350dabda3c10,b60ed2c741306163] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:37:16.613 INFO 26087 --- [ocp-schedules-21,76da350dabda3c10,b60ed2c741306163] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:37:16.614 WARN 26087 --- [ocp-async-10,d11fcdfa1e2196f9,5df7fb3416e691db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:16.614 INFO 26087 --- [ocp-async-9,6d38939eb95eb5dc,d19fdd1275a65229] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:16.618 INFO 26087 --- [ocp-schedules-5,23a69c906e7d0fd6,8e946b90b018853b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:37:16.619 INFO 26087 --- [ocp-schedules-5,23a69c906e7d0fd6,8e946b90b018853b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:37:16.623 INFO 26087 --- [ocp-schedules-5,23a69c906e7d0fd6,8e946b90b018853b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:37:16.623 INFO 26087 --- [ocp-schedules-5,23a69c906e7d0fd6,8e946b90b018853b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:37:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:37:16.623229+08:00, lastProcessTime=2025-04-08T21:37:06.622729+08:00 2025-04-08 21:37:16.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:37:16.623229+08:00 2025-04-08 21:37:16.714 INFO 26087 --- [ocp-schedules-24,a1dc5c99f9bd999d,b30beb5c91a70521] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:37:16.766 INFO 26087 --- [ocp-schedules-3,60d5c175d2a7621a,3c454fde725c5da4] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:37:16.767 INFO 26087 --- [ocp-schedules-3,60d5c175d2a7621a,3c454fde725c5da4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:16.777 ERROR 26087 --- [ocp-schedules-3,60d5c175d2a7621a,3c454fde725c5da4] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:37:16.777 INFO 26087 --- [ocp-schedules-3,60d5c175d2a7621a,3c454fde725c5da4] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:37:16.791 INFO 26087 --- [ocp-schedules-25,5e34f269cb54b2af,db0af0a2019693cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:37:16.815 INFO 26087 --- [ocp-schedules-27,832880ec2d05392e,629e59b42489fcc0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:37:16.815 INFO 26087 --- [ocp-schedules-27,832880ec2d05392e,629e59b42489fcc0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:37:16.819 INFO 26087 --- [ocp-schedules-27,832880ec2d05392e,629e59b42489fcc0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:37:16.819 INFO 26087 --- [ocp-schedules-27,832880ec2d05392e,629e59b42489fcc0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:37:16.823 INFO 26087 --- [ocp-schedules-2,af51237242205ab0,c9fa36fd861d6f8f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:37:16.823 INFO 26087 --- [ocp-schedules-27,832880ec2d05392e,629e59b42489fcc0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:37:16.823 INFO 26087 --- [ocp-schedules-27,832880ec2d05392e,629e59b42489fcc0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:37:16.824 INFO 26087 --- [ocp-schedules-2,af51237242205ab0,c9fa36fd861d6f8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:16.840 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:37:16.841 ERROR 26087 --- [ocp-schedules-2,af51237242205ab0,c9fa36fd861d6f8f] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:37:16.841 INFO 26087 --- [ocp-schedules-2,af51237242205ab0,c9fa36fd861d6f8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:37:16.915 INFO 26087 --- [ocp-schedules-22,b01fa12a3b340436,e183924daeca02fb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:37:17.521 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:17.521 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:17.576 INFO 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:37:17.576 INFO 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:37:17.578 INFO 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:37:17.579 WARN 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:37:17.579 INFO 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:37:17.580 WARN 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:37:17.583 INFO 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:37:17.585 WARN 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:37:17.586 ERROR 26087 --- [ocp-schedules-7,51828583ac915278,4ee73b03ee950fe4] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:37:17.928 INFO 26087 --- [ocp-schedules-24,c674b2776f5ec053,0f718d2119eba6b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:37:17.983 INFO 26087 --- [ocp-schedules-24,c674b2776f5ec053,0f718d2119eba6b0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:37:17.983 INFO 26087 --- [ocp-schedules-24,c674b2776f5ec053,0f718d2119eba6b0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:37:18.271 INFO 26087 --- [prometheus-minute-schedule-1,f3e4cf90b936f5c0,b3e0228c1908cb57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:18.614 WARN 26087 --- [ocp-async-4,94db648ac7e4dcfa,859cd4c96507e6b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:18.614 INFO 26087 --- [ocp-async-5,a42e3123f2b688c5,19a1f9c46259e051] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:18.941 INFO 26087 --- [ocp-schedules-4,7b38332181531035,4b7eee523d9b3c2b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:37:18.942 INFO 26087 --- [ocp-schedules-4,7b38332181531035,4b7eee523d9b3c2b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:37:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:37:20.522 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:20.522 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:20.614 WARN 26087 --- [ocp-async-1,935977ecf9e0f113,113edea437e527e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:20.614 INFO 26087 --- [ocp-async-6,77bc2bcc672728c4,f9dc84e9e73a437f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:21.271 INFO 26087 --- [prometheus-minute-schedule-1,1c2491cea58c8dbe,d6ed4b23fa687c05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:22.001 INFO 26087 --- [ocp-schedules-16,9ab071d098e8899a,46a2aaa162a7a71c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:37:22.614 WARN 26087 --- [ocp-async-2,51be62ef841efb0c,9675ccf9376c1f4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:22.615 INFO 26087 --- [ocp-async-10,c7318040c42f3d09,f0d7d2561c5dedb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:23.522 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:23.523 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:24.271 INFO 26087 --- [prometheus-minute-schedule-1,ac87bf9555088169,aed55c5449e2dc28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:24.615 WARN 26087 --- [ocp-async-10,0f54ac9cc15c48f5,e6d46e37d61540f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:24.615 INFO 26087 --- [ocp-async-7,a7296d0559597254,3facb88249b0f791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:26.442 INFO 26087 --- [ocp-schedules-29,51652085a1f1cd42,9bc9a7515d71a570] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:37:26.442 INFO 26087 --- [ocp-schedules-30,288f214177f437b4,b5b222cdb54d6b74] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:37:26.486 INFO 26087 --- [ocp-async-3,660430c9a55b827f,3a39ec06042a33f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:37:26.502 INFO 26087 --- [ocp-schedules-6,ef12da93ad3cf1f6,4050dea926995ac1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:37:26.523 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:26.524 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:26.615 WARN 26087 --- [ocp-async-5,ea93a6e2011bb9a9,104f58b5e368677c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:26.615 INFO 26087 --- [ocp-async-2,a8cf265dbd483933,5ac8374fd16c54c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:37:26.622969+08:00, lastProcessTime=2025-04-08T21:37:16.623229+08:00 2025-04-08 21:37:26.715 INFO 26087 --- [ocp-schedules-29,77c5a65b9fcc33cf,d67d1fe8e37d7fd6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:37:26.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:37:26.622969+08:00 2025-04-08 21:37:27.271 INFO 26087 --- [prometheus-minute-schedule-1,e7cb3cd2f16082c4,4ef162467ec3eca3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:28.615 INFO 26087 --- [ocp-async-9,af12f34aa969fa05,420939193a8540e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:28.615 WARN 26087 --- [ocp-async-10,faf2c161d38bba30,eda14ed5100cff49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:28.959 INFO 26087 --- [ocp-schedules-3,e3cf3a509a5a9a9b,8a15954167e7d196] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:37:28.960 INFO 26087 --- [ocp-schedules-3,e3cf3a509a5a9a9b,8a15954167e7d196] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:37:29.524 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:29.524 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:30.271 INFO 26087 --- [prometheus-minute-schedule-1,943fb7e3e1e5e8b6,2b234c0ac3753c92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:30.615 WARN 26087 --- [ocp-async-3,a92bc099f9989afc,c70867c7f10b275c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:30.615 INFO 26087 --- [ocp-async-9,83aac17700cb8449,3f6930ef2306dcd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:32.045 INFO 26087 --- [ocp-schedules-17,8fad9b40ec4fd538,528bb47b036a496d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:37:32.524 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:32.525 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:32.615 INFO 26087 --- [ocp-async-1,d0940bb4c4234afd,96e72a26025c358f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:32.615 WARN 26087 --- [ocp-async-6,6787a9c4154ffb17,bcae0526c7cd6ff2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:33.271 INFO 26087 --- [prometheus-minute-schedule-1,906d8fcf0030b7f5,6b049cadd086a324] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:34.615 WARN 26087 --- [ocp-async-8,dd93fa4482ed0671,c4e2221fd2d2524d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:34.615 INFO 26087 --- [ocp-async-2,36038df4d4f49662,61f32d5ceb0d09d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:34.909 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:37:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:37:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:37:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:37:34.911 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:37:34.915 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:37:34.915 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 21:37:35.525 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:35.526 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:36.271 INFO 26087 --- [prometheus-minute-schedule-1,b51aa35f6ecff8f0,3fe83d8107344ea0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:36.486 INFO 26087 --- [ocp-async-4,d27d88869e66309b,c063a7a425745c17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:37:36.615 WARN 26087 --- [ocp-async-7,38407c17eff7a6a5,24b61c93ee03820b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:36.616 INFO 26087 --- [ocp-async-9,cabfc4b23bc16e3c,346a83d3bf7e9985] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:37:36.624461+08:00, lastProcessTime=2025-04-08T21:37:26.622969+08:00 2025-04-08 21:37:36.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:37:36.624461+08:00 2025-04-08 21:37:36.716 INFO 26087 --- [ocp-schedules-3,cdaae06c3c450e84,3c52bc5599d61536] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:37:37.984 INFO 26087 --- [ocp-schedules-16,14999fb30dae9693,80bbb18cf3443b9a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:37:38.007 INFO 26087 --- [ocp-schedules-16,14999fb30dae9693,80bbb18cf3443b9a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:37:38.007 INFO 26087 --- [ocp-schedules-16,14999fb30dae9693,80bbb18cf3443b9a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:37:38.526 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:38.526 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:38.616 WARN 26087 --- [ocp-async-5,237b46d5ef65cd92,f87d47b1777d136b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:38.616 INFO 26087 --- [ocp-async-6,95a6e890097ebb60,8437eefcc44371cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:38.980 INFO 26087 --- [ocp-schedules-28,6a043cd30aac63fa,693d63d71ac1d3f9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:37:38.981 INFO 26087 --- [ocp-schedules-28,6a043cd30aac63fa,693d63d71ac1d3f9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:37:39.271 INFO 26087 --- [prometheus-minute-schedule-1,23dc7e07e630261b,e539d4e87d26b569] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:40.001 INFO 26087 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:37:40.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:37:40.616 WARN 26087 --- [ocp-async-4,ce56256f1646414c,f4d26aa434720520] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:40.616 INFO 26087 --- [ocp-async-10,3998e1cbd1ddf0fa,584f9d38f4287b2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:41.526 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:41.527 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:42.065 INFO 26087 --- [ocp-schedules-9,53dd2301818fb3ab,1c7e58031dfb0c5f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:37:42.271 INFO 26087 --- [prometheus-minute-schedule-1,b47f651b1e903c22,10da52a140e7ca28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:42.616 WARN 26087 --- [ocp-async-3,6ab3f48fe8bb7e45,947c8120c474b3cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:42.616 INFO 26087 --- [ocp-async-2,18427377677d1ab0,b9d3fe51e9f3cde7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:44.527 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:44.528 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:44.616 WARN 26087 --- [ocp-async-5,a2d2e972dad1e3fb,2eff25e69989d2e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:44.616 INFO 26087 --- [ocp-async-1,a6dc437f4ef89b45,b1ffc48a1187e72d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:45.271 INFO 26087 --- [prometheus-minute-schedule-1,27799e521538f2fe,3eeadae8ad3ae132] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:46.437 INFO 26087 --- [ocp-schedules-30,869f295b4e82a3f9,332730261e6f85e6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:37:46.437 INFO 26087 --- [ocp-schedules-8,b920bbee706ec7ce,ffc9bcd80328594a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:37:46.438 INFO 26087 --- [ocp-schedules-8,b920bbee706ec7ce,ffc9bcd80328594a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:37:46.439 INFO 26087 --- [ocp-schedules-2,fa30d83d0d520f4a,c4b3109836f232b5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:37:46.439 INFO 26087 --- [ocp-schedules-2,fa30d83d0d520f4a,c4b3109836f232b5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:37:46.439 INFO 26087 --- [ocp-schedules-13,fd5286ae14bd71e3,2bd60f894923565c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:37:46.439 INFO 26087 --- [ocp-schedules-13,fd5286ae14bd71e3,2bd60f894923565c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:37:46.441 INFO 26087 --- [ocp-schedules-20,097588d50ea920dc,ad95dd4d906eb2cd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:37:46.441 INFO 26087 --- [ocp-schedules-27,da485a57dc78c45e,586d3ea37ae4093d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:37:46.441 INFO 26087 --- [ocp-schedules-20,097588d50ea920dc,ad95dd4d906eb2cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:46.441 INFO 26087 --- [ocp-schedules-27,da485a57dc78c45e,586d3ea37ae4093d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:37:46.445 INFO 26087 --- [ocp-schedules-6,92f7b9daa0d9d0c8,b278b51f45e16127] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:37:46.486 INFO 26087 --- [ocp-async-2,920edd050db097e5,c1881b78cd7c190d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:37:46.496 INFO 26087 --- [ocp-schedules-15,2b3282e2c61c29b5,9a17f5602bd7f56d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:37:46.496 INFO 26087 --- [ocp-schedules-15,e43eb9fe42e1ef00,fb667714cedee825] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:37:46.496 INFO 26087 --- [ocp-schedules-15,e43eb9fe42e1ef00,fb667714cedee825] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:37:46.502 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:37:46.502 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:37:46.572 INFO 26087 --- [ocp-schedules-19,63f9c810ed18271b,d57ed9c7acfc1db9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:37:46.616 WARN 26087 --- [ocp-async-4,ed8086cdf3b92147,c2ebc7b10c157388] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:46.616 INFO 26087 --- [ocp-async-10,102dc33a0dcb2b8e,36e15de83d3c44f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:37:46.623342+08:00, lastProcessTime=2025-04-08T21:37:36.624461+08:00 2025-04-08 21:37:46.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:37:46.623342+08:00 2025-04-08 21:37:46.718 INFO 26087 --- [ocp-schedules-21,291c1558599d7fd2,38f5316a1054a8fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:37:47.528 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:47.528 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:48.271 INFO 26087 --- [prometheus-minute-schedule-1,13d35aaa1e9e6a08,cdb520ea3781576d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:48.616 WARN 26087 --- [ocp-async-3,d4572be33a6b1d5a,afc5095d95974f12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:48.616 INFO 26087 --- [ocp-async-7,d7732e49238cf324,71e1b0f75ddd87f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:48.985 INFO 26087 --- [ocp-schedules-5,9a42c9ea1c6543ec,af1b673deb7c2396] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:37:48.987 INFO 26087 --- [ocp-schedules-5,9a42c9ea1c6543ec,af1b673deb7c2396] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:37:50.529 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:50.529 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:50.617 WARN 26087 --- [ocp-async-1,c68032a393f8d8cf,6ddfd9f383164061] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:50.617 INFO 26087 --- [ocp-async-2,9a72f90135b0abdb,fc55201acf849cb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:51.271 INFO 26087 --- [prometheus-minute-schedule-1,d950d874b4910f0c,70026a2eb7dadf42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:52.093 INFO 26087 --- [ocp-schedules-29,00aa5befc13b19b4,effe959a2b3efe57] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:37:52.617 WARN 26087 --- [ocp-async-6,ecce08db9394b410,222260441ad61e03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:52.617 INFO 26087 --- [ocp-async-4,30cb73af6f8b46a4,7f8c104a79795e96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:53.529 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:53.530 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:54.271 INFO 26087 --- [prometheus-minute-schedule-1,b3304158e5c85798,43a4a0c2ea9a1854] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:54.617 WARN 26087 --- [ocp-async-9,bda3041c5ceb54a5,4c4143b2bb910160] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:54.617 INFO 26087 --- [ocp-async-10,ded6a125278c6c44,851b67552fa76d44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:56.442 INFO 26087 --- [ocp-schedules-27,da485a57dc78c45e,586d3ea37ae4093d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:37:56.442 INFO 26087 --- [ocp-schedules-20,097588d50ea920dc,ad95dd4d906eb2cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:37:56.486 INFO 26087 --- [ocp-async-3,eb5281c2ffafc0ce,eca2122fa505e31c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:37:56.530 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:56.530 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:37:56.617 INFO 26087 --- [ocp-async-5,db09702f1b9fe90e,2aaabb9810e34ee6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:56.617 WARN 26087 --- [ocp-async-1,94db227d23209bae,9a2fce871bb3dabe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:37:56.622500+08:00, lastProcessTime=2025-04-08T21:37:46.623342+08:00 2025-04-08 21:37:56.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:37:56.622500+08:00 2025-04-08 21:37:56.719 INFO 26087 --- [ocp-schedules-6,94ddd68b9a210d10,7bc0d6bef03f75e0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:37:57.271 INFO 26087 --- [prometheus-minute-schedule-1,0b109cd9ea6aecec,2742dbe1552b04d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119480, interval=60 2025-04-08 21:37:58.008 INFO 26087 --- [ocp-schedules-8,8780ca3c5fee1156,3bc7e2daef357a00] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:37:58.042 INFO 26087 --- [ocp-schedules-8,8780ca3c5fee1156,3bc7e2daef357a00] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:37:58.042 INFO 26087 --- [ocp-schedules-8,8780ca3c5fee1156,3bc7e2daef357a00] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:37:58.617 WARN 26087 --- [ocp-async-6,70133ae2bd2cf9ec,0f039ec893ee77ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:37:58.617 INFO 26087 --- [ocp-async-8,ec1df6f13527e846,b3542caf1b90cdb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:37:58.990 INFO 26087 --- [ocp-schedules-22,6d489d1532ae6409,a2416f235d84578f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:37:58.991 INFO 26087 --- [ocp-schedules-22,6d489d1532ae6409,a2416f235d84578f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:37:59.530 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:37:59.531 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:00.271 INFO 26087 --- [prometheus-minute-schedule-1,eb1bdb3c14e491fb,9241b34ca9b3bea4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:00.617 WARN 26087 --- [ocp-async-3,2fa8ea0edff18d65,5c5da10d62547e3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:00.617 INFO 26087 --- [ocp-async-9,5a69d61cb7b551e7,b2d2ae0545fb8bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:02.120 INFO 26087 --- [ocp-schedules-22,eb51265e386f7046,29fa54a19d504424] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:38:02.531 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:02.532 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:02.617 INFO 26087 --- [ocp-async-9,bb59c9433143f521,dd44fcc7c1623f10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:02.617 WARN 26087 --- [ocp-async-1,5bbe6c0b53d48109,65a4b82f1871153e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:03.271 INFO 26087 --- [prometheus-minute-schedule-1,cbc91b93faf64579,d2d50cd816e97860] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:04.617 WARN 26087 --- [ocp-async-9,0521a41d73ef93ee,ec0be994fa021e80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:04.618 INFO 26087 --- [ocp-async-6,0edb9c0b456f46f5,e57ce04b4cf07ae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:05.532 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:05.533 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:06.271 INFO 26087 --- [prometheus-minute-schedule-1,6eb6a797ff0b7030,d110b01cc6442b71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:06.272 INFO 26087 --- [prometheus-cache-monitor1,b8420a6f8e3cce0c,97ddbb45fe3644ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1518,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:38:06.272 INFO 26087 --- [prometheus-cache-monitor1,b8420a6f8e3cce0c,97ddbb45fe3644ae] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:38:06.487 INFO 26087 --- [ocp-async-5,58fc584508fdcab1,ad7d8786c3b32d17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:38:06.618 WARN 26087 --- [ocp-async-3,80f81d0f0e250f7d,d28a9322104fb508] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:06.618 INFO 26087 --- [ocp-async-10,c41168b9479a3d78,dfc8233782a7f561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:38:06.623199+08:00, lastProcessTime=2025-04-08T21:37:56.622500+08:00 2025-04-08 21:38:06.673 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:34:06.593, end=2025-04-08T21:36:06.645} 2025-04-08 21:38:06.676 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:38:06.676 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:38:06.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:38:06.623199+08:00 2025-04-08 21:38:06.721 INFO 26087 --- [ocp-schedules-26,f4641d89873eb6e8,7bebb8a4ca45750b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:38:08.067 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=192.168.1.61:8080, expireInSeconds=30, createTime=2025-04-08T21:03:07+08:00, updateTime=2025-04-08T21:37:37+08:00, expireTime=null, isLeader=false} 2025-04-08 21:38:08.068 WARN 26087 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-04-08 21:38:08.533 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:08.534 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:08.618 WARN 26087 --- [ocp-async-10,c46eb40e0b695a66,d9684b34b2bd808f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:08.618 INFO 26087 --- [ocp-async-2,cb648095e4b3dba3,a308883196f7e258] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:09.007 INFO 26087 --- [ocp-schedules-9,779a1622ee7c23b1,8f17ef209b17d254] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:38:09.008 INFO 26087 --- [ocp-schedules-9,779a1622ee7c23b1,8f17ef209b17d254] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:38:09.271 INFO 26087 --- [prometheus-minute-schedule-1,0dde877d3fcedb8e,d0441b8d22d66a34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:10.002 INFO 26087 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:38:10.002 INFO 26087 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:38:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:38:10.618 WARN 26087 --- [ocp-async-2,be297d84a1cff782,f2978632ee0e29fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:10.618 INFO 26087 --- [ocp-async-4,d30665acda60529c,d8a24742b3ccb0b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:11.534 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:11.534 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:12.145 INFO 26087 --- [ocp-schedules-1,b2239b1290fc878c,80617bf275fc968b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:38:12.271 INFO 26087 --- [prometheus-minute-schedule-1,b86fb79a2e531bab,d0a828bfc07cc5b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:12.443 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119420, nodeCount=38, affectRows=38 2025-04-08 21:38:12.444 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119420, nodeCount=38, validCount=38 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119420, nodeCount=42, affectRows=42 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119420, nodeCount=191, affectRows=191 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119420, nodeCount=56, affectRows=56 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119420, nodeCount=53, affectRows=53 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119420, nodeCount=41, validCount=41 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119420, nodeCount=56, validCount=56 2025-04-08 21:38:12.449 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119420, nodeCount=52, validCount=52 2025-04-08 21:38:12.450 INFO 26087 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119420, nodeCount=190, validCount=190 2025-04-08 21:38:12.618 WARN 26087 --- [ocp-async-3,720d373530fee01d,e6d0e5dc870bbbf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:12.618 INFO 26087 --- [ocp-async-1,7875e646898c3b00,94409f1b1b1027d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:14.534 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:14.535 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:14.618 WARN 26087 --- [ocp-async-10,b18dc757adadbe62,5697d208e2acf9c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:14.618 INFO 26087 --- [ocp-async-7,2b835151cdf9f98e,699100bd36ac0117] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:15.271 INFO 26087 --- [prometheus-minute-schedule-1,a4607a5e4342a0c1,7dca6a0d3747ccd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:16.433 INFO 26087 --- [ocp-schedules-1,532ec338217f2e48,383b5069b30cc83d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:38:16.434 INFO 26087 --- [ocp-schedules-22,5231179f97b06461,e7eecf4b295f53d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:38:16.437 INFO 26087 --- [ocp-schedules-27,784bcff9fafe80df,5e1ffdf99b13366e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:38:16.437 INFO 26087 --- [ocp-schedules-27,784bcff9fafe80df,5e1ffdf99b13366e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:38:16.437 INFO 26087 --- [ocp-schedules-29,0634abcbbf295705,5a170972c6cadb54] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:38:16.440 INFO 26087 --- [ocp-schedules-16,37b1bb2126355301,b1b58fbed80fe983] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:38:16.440 INFO 26087 --- [ocp-schedules-16,37b1bb2126355301,b1b58fbed80fe983] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:38:16.440 INFO 26087 --- [ocp-schedules-26,33fadc03aeef059a,b88afc4b81ef7290] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:38:16.440 INFO 26087 --- [ocp-schedules-26,33fadc03aeef059a,b88afc4b81ef7290] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:38:16.440 INFO 26087 --- [ocp-schedules-22,5231179f97b06461,e7eecf4b295f53d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:38:16.441 INFO 26087 --- [ocp-schedules-8,b57ba57b1017ce4e,ab6f78ff66c46b47] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:38:16.441 INFO 26087 --- [ocp-schedules-10,075e86bff5b6bbfc,b81eacaf658fbea4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:38:16.441 INFO 26087 --- [ocp-schedules-10,075e86bff5b6bbfc,b81eacaf658fbea4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:16.441 INFO 26087 --- [ocp-schedules-8,b57ba57b1017ce4e,ab6f78ff66c46b47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:38:16.446 INFO 26087 --- [ocp-schedules-1,532ec338217f2e48,383b5069b30cc83d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:38:16.453 INFO 26087 --- [ocp-schedules-13,1eee177b88f8b4c1,24153b728c497197] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:38:16.453 INFO 26087 --- [ocp-schedules-13,1eee177b88f8b4c1,24153b728c497197] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:38:16.463 INFO 26087 --- [ocp-schedules-9,a7cbe0078449bb51,40dca0b685f49b39] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:38:16.463 INFO 26087 --- [ocp-schedules-20,a5e515b40fa563d0,704692e73fd5e0bf] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:38:16.464 INFO 26087 --- [ocp-schedules-9,a7cbe0078449bb51,40dca0b685f49b39] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:16.470 INFO 26087 --- [pool-overview-container-2,72cec83d2d6618f0,94933183efdd6de6] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:38:16.470 INFO 26087 --- [pool-overview-container-6,14dfd94e0023b55d,f55475c337a133a2] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:38:16.475 INFO 26087 --- [pool-11-thread-16,14dfd94e0023b55d,0a54355951d4f1ed] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:38:16.475 INFO 26087 --- [pool-11-thread-14,14dfd94e0023b55d,67f4c96af1b08ba7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:38:16.475 INFO 26087 --- [pool-11-thread-15,14dfd94e0023b55d,4675f60d75e9d1af] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:38:16.475 INFO 26087 --- [pool-11-thread-13,14dfd94e0023b55d,245af2ec81f99898] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:38:16.475 INFO 26087 --- [pool-11-thread-15,14dfd94e0023b55d,4675f60d75e9d1af] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:38:16.477 INFO 26087 --- [ForkJoinPool-7-worker-15,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:38:16.477 INFO 26087 --- [ForkJoinPool-7-worker-6,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:38:16.477 INFO 26087 --- [ForkJoinPool-7-worker-6,14dfd94e0023b55d,451818be9dc8873a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 21:38:16.477 INFO 26087 --- [ForkJoinPool-7-worker-15,14dfd94e0023b55d,4e1516a241b0c7ef] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000003, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:38:16.477 INFO 26087 --- [ForkJoinPool-7-worker-6,14dfd94e0023b55d,451818be9dc8873a] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000003, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:38:16.477 INFO 26087 --- [ForkJoinPool-7-worker-6,14dfd94e0023b55d,451818be9dc8873a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 21:38:16.477 INFO 26087 --- [pool-11-thread-16,14dfd94e0023b55d,0a54355951d4f1ed] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:38:16.477 INFO 26087 --- [pool-11-thread-14,14dfd94e0023b55d,67f4c96af1b08ba7] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:38:16.477 INFO 26087 --- [pool-11-thread-13,14dfd94e0023b55d,245af2ec81f99898] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:38:16.477 INFO 26087 --- [pool-11-thread-13,14dfd94e0023b55d,245af2ec81f99898] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:38:16.478 INFO 26087 --- [pool-11-thread-15,14dfd94e0023b55d,4675f60d75e9d1af] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:38:16.478 INFO 26087 --- [pool-11-thread-15,14dfd94e0023b55d,4675f60d75e9d1af] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:38:16.487 INFO 26087 --- [ocp-async-2,3d8df33bdc37df9d,3a893f262e0b895c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:38:16.502 INFO 26087 --- [ocp-schedules-15,e43eb9fe42e1ef00,fb667714cedee825] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:38:16.502 INFO 26087 --- [ocp-schedules-15,56838e72cc0fce76,127a97df7e57bd1d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:38:16.503 INFO 26087 --- [ocp-schedules-15,56838e72cc0fce76,127a97df7e57bd1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:16.507 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:38:16.507 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:38:16.533 INFO 26087 --- [ocp-schedules-6,6e49beeaa72cdca9,5033171064190273] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:38:16.568 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:38:16.568 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:38:16.573 INFO 26087 --- [ocp-schedules-22,1408c2f6bd6d0f25,0bab6b4d8425941f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:38:16.573 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:38:16.573 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:38:16.573 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:38:16.574 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:38:16.574 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:38:16.574 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:38:16.576 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:38:16.576 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:38:16.576 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:38:16.577 INFO 26087 --- [ocp-schedules-5,1a9d6eb806548c4e,07b7bc2eef2c2163] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:38:16.577 INFO 26087 --- [ocp-schedules-5,1a9d6eb806548c4e,07b7bc2eef2c2163] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:38:16.579 INFO 26087 --- [ocp-schedules-16,7204dc42da9434d3,932258677483325c] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:38:16.588 INFO 26087 --- [ocp-schedules-13,b91c3c6af5344587,f178817a0a9134a3] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:38:16.588 INFO 26087 --- [ocp-schedules-13,b91c3c6af5344587,f178817a0a9134a3] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:38:16.602 INFO 26087 --- [ocp-schedules-1,8e0981707596413b,89f1457f4e541992] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:38:16.602+08:00, clusters=[1000003] 2025-04-08 21:38:16.602 INFO 26087 --- [ocp-schedules-20,9de543644aeb4b9c,6113a72dc0aaa43d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:38:16.603 INFO 26087 --- [ocp-schedules-1,8e0981707596413b,89f1457f4e541992] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:38:16.606 INFO 26087 --- [ocp-schedules-19,7a5a206329619d76,91a5615a8972730b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:38:16.606+08:00, clusters=[1000003] 2025-04-08 21:38:16.609 INFO 26087 --- [ocp-schedules-19,7a5a206329619d76,91a5615a8972730b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:38:16.614 INFO 26087 --- [ocp-schedules-23,1770f83abeb80e7d,8b054693dde8fb73] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:38:16.615 INFO 26087 --- [ocp-schedules-23,1770f83abeb80e7d,8b054693dde8fb73] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:38:16.615 INFO 26087 --- [ocp-schedules-23,1770f83abeb80e7d,8b054693dde8fb73] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:38:16.618 WARN 26087 --- [ocp-async-6,bfb63f9bf2ac0a47,83e436a4570b76c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:16.618 INFO 26087 --- [ocp-async-4,8446672ecbc93ee2,587c89898597762a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:38:16.622599+08:00, lastProcessTime=2025-04-08T21:38:06.623199+08:00 2025-04-08 21:38:16.623 INFO 26087 --- [ocp-schedules-17,cbb0869afd899dd1,c734c2efece72389] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:38:16.623 INFO 26087 --- [ocp-schedules-17,cbb0869afd899dd1,c734c2efece72389] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:38:16.627 INFO 26087 --- [ocp-schedules-17,cbb0869afd899dd1,c734c2efece72389] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:38:16.627 INFO 26087 --- [ocp-schedules-17,cbb0869afd899dd1,c734c2efece72389] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:38:16.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:38:16.622599+08:00 2025-04-08 21:38:16.722 INFO 26087 --- [ocp-schedules-11,19f3aa2739388355,2efc55768653d7b2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:38:16.745 INFO 26087 --- [ocp-schedules-4,bc073d47882e5563,d6c87532c9435dc8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:38:16.745 INFO 26087 --- [ocp-schedules-4,bc073d47882e5563,d6c87532c9435dc8] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:38:16.750 INFO 26087 --- [pool-29-thread-1,bc073d47882e5563,3c4b38a879441756] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:38:16.755 WARN 26087 --- [ocp-schedules-4,bc073d47882e5563,d6c87532c9435dc8] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$14662e73.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:38:16.755 INFO 26087 --- [ocp-schedules-4,bc073d47882e5563,d6c87532c9435dc8] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:38:16.791 INFO 26087 --- [ocp-schedules-12,54b9c39aab97aa68,641233722a631290] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:38:16.792 INFO 26087 --- [ocp-schedules-12,54b9c39aab97aa68,641233722a631290] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:16.804 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:38:16.804 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:38:16.826 INFO 26087 --- [ocp-schedules-2,2e2ecb3039aa834e,c8027dbb46841a5b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:38:16.826 INFO 26087 --- [ocp-schedules-2,2e2ecb3039aa834e,c8027dbb46841a5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:38:16.826 INFO 26087 --- [ocp-schedules-2,2e2ecb3039aa834e,c8027dbb46841a5b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:38:16.826 INFO 26087 --- [ocp-schedules-2,2e2ecb3039aa834e,c8027dbb46841a5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:38:16.826 INFO 26087 --- [ocp-schedules-2,2e2ecb3039aa834e,c8027dbb46841a5b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:38:16.826 INFO 26087 --- [ocp-schedules-2,2e2ecb3039aa834e,c8027dbb46841a5b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:38:16.841 INFO 26087 --- [ocp-schedules-24,bba528b1a51c103b,579582e41ebc36df] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:38:16.842 INFO 26087 --- [ocp-schedules-24,bba528b1a51c103b,579582e41ebc36df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:16.857 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:38:16.858 ERROR 26087 --- [ocp-schedules-24,bba528b1a51c103b,579582e41ebc36df] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:38:16.858 INFO 26087 --- [ocp-schedules-24,bba528b1a51c103b,579582e41ebc36df] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:38:16.921 INFO 26087 --- [ocp-schedules-29,d250490ed3b6e09c,eb4beeb79a2510d6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:38:17.081 INFO 26087 --- [ocp-schedules-22,89716b9e1f7ae865,17cdb9b33e03e59e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:38:17.087 INFO 26087 --- [ocp-schedules-22,89716b9e1f7ae865,17cdb9b33e03e59e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:38:17.088 WARN 26087 --- [ocp-schedules-22,89716b9e1f7ae865,17cdb9b33e03e59e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:38:17.088 INFO 26087 --- [ocp-schedules-22,89716b9e1f7ae865,17cdb9b33e03e59e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:38:17.089 WARN 26087 --- [ocp-schedules-22,89716b9e1f7ae865,17cdb9b33e03e59e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:38:17.092 INFO 26087 --- [ocp-schedules-22,89716b9e1f7ae865,17cdb9b33e03e59e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=7 2025-04-08 21:38:17.535 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:17.536 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:17.588 INFO 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:38:17.588 INFO 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:38:17.591 INFO 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:38:17.592 WARN 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:38:17.592 INFO 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:38:17.593 WARN 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:38:17.596 INFO 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:38:17.597 WARN 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:38:17.598 ERROR 26087 --- [ocp-schedules-7,c81a902d3ed79136,be391add635a357a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:38:18.043 INFO 26087 --- [ocp-schedules-25,da7e403c37c6d25e,c65dfaee0f800a5b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:38:18.047 INFO 26087 --- [ocp-schedules-25,da7e403c37c6d25e,c65dfaee0f800a5b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:38:18.047 INFO 26087 --- [ocp-schedules-25,da7e403c37c6d25e,c65dfaee0f800a5b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:38:18.271 INFO 26087 --- [prometheus-minute-schedule-1,bc3aa9271231b41b,02b8e3b8433b5afd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:18.619 WARN 26087 --- [ocp-async-4,8a7a9508f186b346,d1a021df47e7df87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:18.619 INFO 26087 --- [ocp-async-9,1dec84b9d4406b21,c67c69117fbe8026] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:19.011 INFO 26087 --- [ocp-schedules-11,4b3bf8b69e088408,b4c0ddbfbd388e43] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:38:19.012 INFO 26087 --- [ocp-schedules-11,4b3bf8b69e088408,b4c0ddbfbd388e43] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:38:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:38:20.536 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:20.536 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:20.619 WARN 26087 --- [ocp-async-7,8cc6e1c402627660,631a5fcd4a256580] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:20.619 INFO 26087 --- [ocp-async-5,b8680a464df2a9ee,acb4b8529024b1d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:21.271 INFO 26087 --- [prometheus-minute-schedule-1,fc56404878129450,feb6d7eea9acd245] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:22.166 INFO 26087 --- [ocp-schedules-24,d4783ef7679b6989,89fcc3f02b70f060] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:38:22.619 WARN 26087 --- [ocp-async-5,fb3245a16c59ee14,95cdbcd71b08cac6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:22.619 INFO 26087 --- [ocp-async-6,79b70ba8dcfc90d0,0dd434769bc5bf92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:23.537 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:23.537 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:24.271 INFO 26087 --- [prometheus-minute-schedule-1,484c159acd6fa083,821736db3964a666] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:24.619 WARN 26087 --- [ocp-async-10,8a2cf9adaab6ff22,e2ee72df66f23a4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:24.619 INFO 26087 --- [ocp-async-3,37bec94783e8b6e8,1a40ad5f460596c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:26.442 INFO 26087 --- [ocp-schedules-10,075e86bff5b6bbfc,b81eacaf658fbea4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:38:26.442 INFO 26087 --- [ocp-schedules-8,b57ba57b1017ce4e,ab6f78ff66c46b47] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:38:26.464 INFO 26087 --- [ocp-schedules-9,a7cbe0078449bb51,40dca0b685f49b39] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 21:38:26.487 INFO 26087 --- [ocp-async-3,83b5fe1157232b90,83f30440019a62cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:38:26.502 INFO 26087 --- [ocp-schedules-25,fa555d8458651af2,b2ff5c27f0d8f439] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:38:26.503 INFO 26087 --- [ocp-schedules-25,fa555d8458651af2,b2ff5c27f0d8f439] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:26.503 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:38:26.537 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:26.538 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:26.619 WARN 26087 --- [ocp-async-5,1efc639854d569ce,d77d4e7049d2de90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:26.619 INFO 26087 --- [ocp-async-2,3db9e17d665417db,05ebed1dbdc3c72e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:26.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:38:26.624386+08:00, lastProcessTime=2025-04-08T21:38:16.622599+08:00 2025-04-08 21:38:26.723 INFO 26087 --- [ocp-schedules-10,01a5f4130607c8e0,45a2fff4166fc7ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:38:26.727 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:38:26.624386+08:00 2025-04-08 21:38:27.271 INFO 26087 --- [prometheus-minute-schedule-1,e0948715f2e86c73,d47a79ae4e4140d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:28.619 WARN 26087 --- [ocp-async-10,faa2a41f1b9b3452,a215585ca73b4fd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:28.619 INFO 26087 --- [ocp-async-6,4af993ba1cc1e30e,e3e1bf65f1d481ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:29.018 INFO 26087 --- [ocp-schedules-18,5e9f61c6cc5216ad,99e89b4f33d26337] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:38:29.019 INFO 26087 --- [ocp-schedules-18,5e9f61c6cc5216ad,99e89b4f33d26337] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:38:29.538 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:29.539 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:30.271 INFO 26087 --- [prometheus-minute-schedule-1,52407adea518a044,4271471a65991964] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:30.619 WARN 26087 --- [ocp-async-3,6dcf8e3a3cfc1c96,cb5570a0e944e00c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:30.619 INFO 26087 --- [ocp-async-4,71ea639ef7dd8256,5071b05d1f5d0355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:32.192 INFO 26087 --- [ocp-schedules-5,c3e5a8cda29afc65,ff6ec8d4e4280071] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:38:32.539 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:32.539 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:32.619 WARN 26087 --- [ocp-async-4,a85ed0dd41a4c1bb,f57e12cee25e0cfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:32.620 INFO 26087 --- [ocp-async-5,a12fb84912ec9703,8162666508cc69c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:33.271 INFO 26087 --- [prometheus-minute-schedule-1,e7d2960c0084c050,435574c120fd5a6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:34.620 WARN 26087 --- [ocp-async-10,8eba2e7c37c8105e,26ac8608abc0b541] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:34.620 INFO 26087 --- [ocp-async-5,f54a35a99beee30e,07960f4603a11ba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:35.539 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:35.540 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:36.271 INFO 26087 --- [prometheus-minute-schedule-1,63aae9d64540f9f0,63bca5ea7b85de40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:36.487 INFO 26087 --- [ocp-async-6,84a108df64853dd4,6fdc2707f4d10242] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:38:36.503 INFO 26087 --- [ocp-schedules-25,fa555d8458651af2,b2ff5c27f0d8f439] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:38:36.620 WARN 26087 --- [ocp-async-3,b04424ded31508bd,4bd57a9f30e640cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:36.620 INFO 26087 --- [ocp-async-9,ea98dd8424953111,f092f6483edef5a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:38:36.622974+08:00, lastProcessTime=2025-04-08T21:38:26.624386+08:00 2025-04-08 21:38:36.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:38:36.622974+08:00 2025-04-08 21:38:36.724 INFO 26087 --- [ocp-schedules-30,ded4715b521e0d87,5c366eddce5339b7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:38:38.049 INFO 26087 --- [ocp-schedules-7,4f6c9d69c2de016c,de2420538d25f3da] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:38:38.069 INFO 26087 --- [ocp-schedules-7,4f6c9d69c2de016c,de2420538d25f3da] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:38:38.069 INFO 26087 --- [ocp-schedules-7,4f6c9d69c2de016c,de2420538d25f3da] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:38:38.540 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:38.541 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:38.620 WARN 26087 --- [ocp-async-9,33bd86091f3ce978,8407199f4e52c06e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:38.620 INFO 26087 --- [ocp-async-1,6f9ad7e087840e0f,58f62228f9a99d57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:39.029 INFO 26087 --- [ocp-schedules-9,b79a9cfcfe512ae9,de04746c79b725a8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:38:39.030 INFO 26087 --- [ocp-schedules-9,b79a9cfcfe512ae9,de04746c79b725a8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:38:39.271 INFO 26087 --- [prometheus-minute-schedule-1,e730af253b1dae99,a576cee8f57d4361] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:40.001 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:38:40.002 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:38:40.620 WARN 26087 --- [ocp-async-2,4ade0ccb51bc2b2b,17872466570b3cd4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:40.620 INFO 26087 --- [ocp-async-6,ec65617bba1294a6,bef030866598df67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:41.541 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:41.541 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:42.212 INFO 26087 --- [ocp-schedules-14,b43c84acab04dc86,a3ac1347ce74db1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:38:42.271 INFO 26087 --- [prometheus-minute-schedule-1,a84d0142d876cf16,5127e699389bc13e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:42.620 WARN 26087 --- [ocp-async-7,f7cb14b0e3649e11,87756a34c3eeb1b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:42.620 INFO 26087 --- [ocp-async-5,da8934da76d25b34,1186b55e914224d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:44.541 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:44.542 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:44.620 WARN 26087 --- [ocp-async-9,40f60d556f55a4f0,d10a0d8adc9aea85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:44.620 INFO 26087 --- [ocp-async-8,2652818e5fb6e87b,121e658314d60d56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:45.271 INFO 26087 --- [prometheus-minute-schedule-1,a95d297b3e900c27,7f88704625cd29e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:46.437 INFO 26087 --- [ocp-schedules-11,80b544a239b24393,c81742ef8341e0b8] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:38:46.437 INFO 26087 --- [ocp-schedules-11,80b544a239b24393,c81742ef8341e0b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:38:46.437 INFO 26087 --- [ocp-schedules-11,50ddbd72776f83a1,b586a01c1fbdbf4e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:38:46.439 INFO 26087 --- [ocp-schedules-20,f81ee8a3c4c05f80,66b812a9d0e2b2f6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:38:46.439 INFO 26087 --- [ocp-schedules-20,f81ee8a3c4c05f80,66b812a9d0e2b2f6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:38:46.439 INFO 26087 --- [ocp-schedules-19,fc638cbff36d0c25,dbe9907dd94c37e0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:38:46.440 INFO 26087 --- [ocp-schedules-19,fc638cbff36d0c25,dbe9907dd94c37e0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:38:46.440 INFO 26087 --- [ocp-schedules-7,4bdd0068832b4b89,bccfc0a97e7e64a3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:38:46.441 INFO 26087 --- [ocp-schedules-23,6c8eba19a6a60d85,9fd2b3db9e963cca] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:38:46.441 INFO 26087 --- [ocp-schedules-29,09a40789eca4fb5b,f995c6343ef66f44] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:38:46.441 INFO 26087 --- [ocp-schedules-23,6c8eba19a6a60d85,9fd2b3db9e963cca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:46.441 INFO 26087 --- [ocp-schedules-29,09a40789eca4fb5b,f995c6343ef66f44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:38:46.487 INFO 26087 --- [ocp-async-5,e3468a31ceaf841e,c5bd68cc594bd61d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:38:46.508 INFO 26087 --- [ocp-schedules-15,56838e72cc0fce76,127a97df7e57bd1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:38:46.508 INFO 26087 --- [ocp-schedules-15,75d480ad5bc34bcf,ab5521523a3fbf1f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:38:46.508 INFO 26087 --- [ocp-schedules-15,75d480ad5bc34bcf,ab5521523a3fbf1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:38:46.513 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:38:46.513 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:38:46.574 INFO 26087 --- [ocp-schedules-2,77ce87c1e6667be4,f41114431ee3f716] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:38:46.621 WARN 26087 --- [ocp-async-2,72225b0475e38120,74d63c8f62a1d15a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:46.621 INFO 26087 --- [ocp-async-10,4a193ca6da49f9cb,24b615aaf665a26e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:38:46.623263+08:00, lastProcessTime=2025-04-08T21:38:36.622974+08:00 2025-04-08 21:38:46.700 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:38:46.623263+08:00 2025-04-08 21:38:46.726 INFO 26087 --- [ocp-schedules-8,b27a42acd80da8eb,939acf2fc2fbbea1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:38:47.542 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:47.543 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:48.271 INFO 26087 --- [prometheus-minute-schedule-1,c0c41936f70678f9,7dab720dde2fc3f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:48.621 WARN 26087 --- [ocp-async-4,a0a666b5612bd390,0667732297ef3723] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:48.621 INFO 26087 --- [ocp-async-7,9c9a5cc81f1e76dd,574679a89fc1dfb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:49.043 INFO 26087 --- [ocp-schedules-25,19a3cd02c66ffca5,de006087e0ad12c1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:38:49.044 INFO 26087 --- [ocp-schedules-25,19a3cd02c66ffca5,de006087e0ad12c1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:38:50.543 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:50.543 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:50.621 WARN 26087 --- [ocp-async-9,066269f84483a96c,60392533d11a9e9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:50.621 INFO 26087 --- [ocp-async-5,4ee3949d2f51bb50,e982e52670058196] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:51.271 INFO 26087 --- [prometheus-minute-schedule-1,7881b497753501d4,1c07db8271f062b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:52.238 INFO 26087 --- [ocp-schedules-3,e5507ad44a3e1397,62cff14eb61dc7b2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:38:52.621 WARN 26087 --- [ocp-async-6,748fcd2ee15f09d7,a7d835c03e744c73] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:52.621 INFO 26087 --- [ocp-async-2,902753287a89f082,3e2ed961e5781650] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:53.544 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:53.544 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:54.271 INFO 26087 --- [prometheus-minute-schedule-1,25fad65660359372,574578798cdef792] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:54.621 WARN 26087 --- [ocp-async-4,312bedfa8ac07573,34cc9078a2171196] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:54.621 INFO 26087 --- [ocp-async-3,d47016327575422f,75c2c5e7581c84e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:56.442 INFO 26087 --- [ocp-schedules-23,6c8eba19a6a60d85,9fd2b3db9e963cca] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:38:56.442 INFO 26087 --- [ocp-schedules-29,09a40789eca4fb5b,f995c6343ef66f44] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:38:56.487 INFO 26087 --- [ocp-async-2,930d6caf6c2d198b,ad0543c023bd727c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:38:56.544 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:56.545 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:38:56.621 WARN 26087 --- [ocp-async-8,d707623b3c0c3969,0fe72cad770e0e70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:56.621 INFO 26087 --- [ocp-async-9,763c41a47b1cf0cd,80494decc6fe5c9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:38:56.623646+08:00, lastProcessTime=2025-04-08T21:38:46.623263+08:00 2025-04-08 21:38:56.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:38:56.623646+08:00 2025-04-08 21:38:56.727 INFO 26087 --- [ocp-schedules-22,b93745298a13770c,b9c328a84d6b4eb0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:38:57.271 INFO 26087 --- [prometheus-minute-schedule-1,29e4041a758b7032,1bb83e211063441b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119540, interval=60 2025-04-08 21:38:58.070 INFO 26087 --- [ocp-schedules-14,0616e20d1ecfa2ad,cb5b8a882a24ee96] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:38:58.078 INFO 26087 --- [ocp-schedules-14,0616e20d1ecfa2ad,cb5b8a882a24ee96] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:38:58.078 INFO 26087 --- [ocp-schedules-14,0616e20d1ecfa2ad,cb5b8a882a24ee96] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:38:58.621 WARN 26087 --- [ocp-async-6,fe6d72e2b241c755,ecd3604ef078ba05] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:38:58.621 INFO 26087 --- [ocp-async-1,832f3ae0c1843fa1,f304ac527e13bd18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:38:59.075 INFO 26087 --- [ocp-schedules-6,7b767c525e5b9fad,f4110bb41aa1f4c9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:38:59.077 INFO 26087 --- [ocp-schedules-6,7b767c525e5b9fad,f4110bb41aa1f4c9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:38:59.545 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:38:59.545 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:00.271 INFO 26087 --- [prometheus-minute-schedule-1,e25170ee681f8cbe,77823b306c469f4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:00.621 WARN 26087 --- [ocp-async-3,e82363e14671c2f5,55f48fec5aba5fd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:00.621 INFO 26087 --- [ocp-async-4,7e20c09ba06cc755,fe1986f5039f1499] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:02.259 INFO 26087 --- [ocp-schedules-16,0448bfb23c7871a2,267b620adfede3f4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:39:02.545 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:02.546 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:02.622 WARN 26087 --- [ocp-async-4,dc8f0c23201fade6,24c20971f0edefae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:02.622 INFO 26087 --- [ocp-async-7,23889aacc8f36612,3ab63a10d456d354] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:03.271 INFO 26087 --- [prometheus-minute-schedule-1,5b2b004793ad263b,5dc4ccd0869974c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:04.622 WARN 26087 --- [ocp-async-7,fe7912c1d054d58a,3843da6764184ece] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:04.622 INFO 26087 --- [ocp-async-9,633839c61cd4f7e3,037e0adb4448f40d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:05.546 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:05.547 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:06.271 INFO 26087 --- [prometheus-minute-schedule-1,c834220b6cef676f,312612b359a599c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:06.272 INFO 26087 --- [prometheus-cache-monitor1,ef8b74d329a78764,fbb9409bdcf0d8ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:39:06.272 INFO 26087 --- [prometheus-cache-monitor1,ef8b74d329a78764,fbb9409bdcf0d8ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:39:06.356 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:33:06.279, end=2025-04-08T21:38:06.337} 2025-04-08 21:39:06.358 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:39:06.359 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:39:06.361 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:33:06.28, end=2025-04-08T21:38:06.34} 2025-04-08 21:39:06.363 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:39:06.363 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:39:06.488 INFO 26087 --- [ocp-async-6,941b9b664a51070c,d4e2c1670c311ce7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:39:06.622 INFO 26087 --- [ocp-async-9,bb42d3aaae445963,c97846ad26dfef6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:06.622 WARN 26087 --- [ocp-async-2,9405929d1157d1d3,c74839bbb2a6ae47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:39:06.623775+08:00, lastProcessTime=2025-04-08T21:38:56.623646+08:00 2025-04-08 21:39:06.729 INFO 26087 --- [ocp-schedules-2,4b01344220b77b3f,f1917b77658e99b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:39:06.733 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:39:06.623775+08:00 2025-04-08 21:39:08.547 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:08.547 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:08.622 WARN 26087 --- [ocp-async-5,b5a662d2d36c0c00,851a8d84210769e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:08.622 INFO 26087 --- [ocp-async-9,925030b85691c449,73de8d6147b5d432] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:09.089 INFO 26087 --- [ocp-schedules-24,75a2ae7880361dea,b70355ffd198bf90] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:39:09.090 INFO 26087 --- [ocp-schedules-24,75a2ae7880361dea,b70355ffd198bf90] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:39:09.271 INFO 26087 --- [prometheus-minute-schedule-1,be3c6835911f8d6f,b7edf5dd51af682b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:10.002 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:39:10.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:39:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:39:10.622 WARN 26087 --- [ocp-async-3,0f2b68966b3c5bb4,2a8b2b5125a4845b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:10.622 INFO 26087 --- [ocp-async-9,bdd45e1c21b835d0,01c1d5c9ffb70b0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:11.547 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:11.548 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:12.271 INFO 26087 --- [prometheus-minute-schedule-1,abb5df0775b77ae7,db27ed1a8550a30f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:12.284 INFO 26087 --- [ocp-schedules-5,eff1e0f00bdfec3f,aec26b04005f84c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:39:12.463 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119480, nodeCount=124, affectRows=124 2025-04-08 21:39:12.464 INFO 26087 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119480, nodeCount=123, validCount=123 2025-04-08 21:39:12.473 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119480, nodeCount=134, affectRows=134 2025-04-08 21:39:12.473 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119480, nodeCount=122, affectRows=122 2025-04-08 21:39:12.474 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119480, nodeCount=121, validCount=121 2025-04-08 21:39:12.474 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119480, nodeCount=133, validCount=133 2025-04-08 21:39:12.622 WARN 26087 --- [ocp-async-1,eee8e412c978d7de,a369d090e8514313] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:12.622 INFO 26087 --- [ocp-async-2,ee9fe0c7e6d2308c,64d5517baf0af70d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:14.548 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:14.549 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:14.622 WARN 26087 --- [ocp-async-2,165146750cc90048,72a74c0da2caa9ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:14.622 INFO 26087 --- [ocp-async-5,d5f486b0506b4f5c,26bc8566d431e97b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:15.271 INFO 26087 --- [prometheus-minute-schedule-1,e4613a34b1d83830,a5f3c531928d3ff0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:16.433 INFO 26087 --- [ocp-schedules-19,727f2703783cfdc4,59634ded0995bc10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:39:16.434 INFO 26087 --- [ocp-schedules-21,c3066cd75b68a8bd,82fad57866fb04a6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:39:16.437 INFO 26087 --- [ocp-schedules-16,b5bcde79d5adc0a9,cfaa3d9e5998178e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:39:16.437 INFO 26087 --- [ocp-schedules-16,b5bcde79d5adc0a9,cfaa3d9e5998178e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:39:16.437 INFO 26087 --- [ocp-schedules-27,b75fe8016424b151,36b3c34546fc19fc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:39:16.439 INFO 26087 --- [ocp-schedules-28,5d533f071ba767b0,8c600c1b701b4cbf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:39:16.439 INFO 26087 --- [ocp-schedules-5,f8e773218bb38437,42b8a18307fba5e3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:39:16.439 INFO 26087 --- [ocp-schedules-5,f8e773218bb38437,42b8a18307fba5e3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:39:16.439 INFO 26087 --- [ocp-schedules-28,5d533f071ba767b0,8c600c1b701b4cbf] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:39:16.441 INFO 26087 --- [ocp-schedules-9,9a4bb7395a11b7c6,33c0f0a27219ac46] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:39:16.441 INFO 26087 --- [ocp-schedules-7,7f149f0cb5ae853e,929f8315dd5eabb9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:39:16.441 INFO 26087 --- [ocp-schedules-21,c3066cd75b68a8bd,82fad57866fb04a6] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:39:16.441 INFO 26087 --- [ocp-schedules-9,9a4bb7395a11b7c6,33c0f0a27219ac46] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:16.441 INFO 26087 --- [ocp-schedules-7,7f149f0cb5ae853e,929f8315dd5eabb9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:39:16.456 INFO 26087 --- [ocp-schedules-19,727f2703783cfdc4,59634ded0995bc10] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:39:16.463 INFO 26087 --- [ocp-schedules-10,d7257bf8e8e9d691,c1ff6ad92b7d0dae] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:39:16.488 INFO 26087 --- [ocp-async-3,9697f36d45dac500,e2148195308ad830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:39:16.505 INFO 26087 --- [ocp-schedules-8,43f7f395ad0c4744,be5846b33f8d48d3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:39:16.505 INFO 26087 --- [ocp-schedules-8,43f7f395ad0c4744,be5846b33f8d48d3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:39:16.513 INFO 26087 --- [ocp-schedules-15,75d480ad5bc34bcf,ab5521523a3fbf1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:39:16.513 INFO 26087 --- [ocp-schedules-15,5d03e9a43cbc5e46,7dd8a326a703d878] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:39:16.514 INFO 26087 --- [ocp-schedules-15,5d03e9a43cbc5e46,7dd8a326a703d878] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:16.519 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:39:16.519 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:39:16.534 INFO 26087 --- [ocp-schedules-14,de9d14ff40d3e9ba,eccbb29f76334075] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:39:16.575 INFO 26087 --- [ocp-schedules-2,79eef53658192db9,bb1f14890bb63023] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:39:16.579 INFO 26087 --- [ocp-schedules-16,a849d3d47f5b9c78,ed884f4554cd7cf0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:39:16.579 INFO 26087 --- [ocp-schedules-16,a849d3d47f5b9c78,ed884f4554cd7cf0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:39:16.590 INFO 26087 --- [ocp-schedules-27,90901824657d8a88,15ab24105dee917a] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:39:16.590 INFO 26087 --- [ocp-schedules-27,90901824657d8a88,15ab24105dee917a] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:39:16.608 INFO 26087 --- [ocp-schedules-28,e06dde2d2ad28c9f,0d0f7f5372739f6c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:39:16.608+08:00, clusters=[1000003] 2025-04-08 21:39:16.608 INFO 26087 --- [ocp-schedules-5,687275c6bcbc976f,4c4b616c5e536fea] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:39:16.608 INFO 26087 --- [ocp-schedules-28,e06dde2d2ad28c9f,0d0f7f5372739f6c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:39:16.613 INFO 26087 --- [ocp-schedules-21,75b670be3b9c97b3,bfdb7b2a3d8b1a68] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:39:16.613+08:00, clusters=[1000003] 2025-04-08 21:39:16.615 INFO 26087 --- [ocp-schedules-21,75b670be3b9c97b3,bfdb7b2a3d8b1a68] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:39:16.615 INFO 26087 --- [ocp-schedules-26,0039c829068827f0,c40f6a5edc8f4cb8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:39:16.617 INFO 26087 --- [ocp-schedules-26,0039c829068827f0,c40f6a5edc8f4cb8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:39:16.617 INFO 26087 --- [ocp-schedules-26,0039c829068827f0,c40f6a5edc8f4cb8] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:39:16.623 INFO 26087 --- [ocp-async-10,aabdff64df6d3c05,928c834029ef8f54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:16.623 WARN 26087 --- [ocp-async-4,5f38f4a5573aacd0,8fbc9262823234a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:39:16.623534+08:00, lastProcessTime=2025-04-08T21:39:06.623775+08:00 2025-04-08 21:39:16.627 INFO 26087 --- [ocp-schedules-29,720fff51e4626a46,651ddc00b6bd4c3d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:39:16.627 INFO 26087 --- [ocp-schedules-29,720fff51e4626a46,651ddc00b6bd4c3d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:39:16.631 INFO 26087 --- [ocp-schedules-29,720fff51e4626a46,651ddc00b6bd4c3d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:39:16.631 INFO 26087 --- [ocp-schedules-29,720fff51e4626a46,651ddc00b6bd4c3d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:39:16.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:39:16.623534+08:00 2025-04-08 21:39:16.730 INFO 26087 --- [ocp-schedules-19,0d9c9dcc783c07cc,f717c10fe8c60b28] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:39:16.778 INFO 26087 --- [ocp-schedules-17,0895ebcde2f6de35,0e39c8cfabbaf693] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:39:16.779 INFO 26087 --- [ocp-schedules-17,0895ebcde2f6de35,0e39c8cfabbaf693] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:16.790 ERROR 26087 --- [ocp-schedules-17,0895ebcde2f6de35,0e39c8cfabbaf693] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:39:16.790 INFO 26087 --- [ocp-schedules-17,0895ebcde2f6de35,0e39c8cfabbaf693] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:39:16.809 INFO 26087 --- [ocp-schedules-12,54b9c39aab97aa68,641233722a631290] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:39:16.827 INFO 26087 --- [ocp-schedules-4,ee482a6a43171a9c,e443d95e49fa5d7d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:39:16.827 INFO 26087 --- [ocp-schedules-4,ee482a6a43171a9c,e443d95e49fa5d7d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:39:16.827 INFO 26087 --- [ocp-schedules-4,ee482a6a43171a9c,e443d95e49fa5d7d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:39:16.827 INFO 26087 --- [ocp-schedules-4,ee482a6a43171a9c,e443d95e49fa5d7d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:39:16.828 INFO 26087 --- [ocp-schedules-4,ee482a6a43171a9c,e443d95e49fa5d7d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:39:16.828 INFO 26087 --- [ocp-schedules-4,ee482a6a43171a9c,e443d95e49fa5d7d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:39:16.858 INFO 26087 --- [ocp-schedules-18,347bb5ebce1ea812,0dfd2c964b40ed02] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:39:16.859 INFO 26087 --- [ocp-schedules-18,347bb5ebce1ea812,0dfd2c964b40ed02] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:16.875 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:39:16.876 ERROR 26087 --- [ocp-schedules-18,347bb5ebce1ea812,0dfd2c964b40ed02] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:39:16.876 INFO 26087 --- [ocp-schedules-18,347bb5ebce1ea812,0dfd2c964b40ed02] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:39:16.927 INFO 26087 --- [ocp-schedules-11,0ef22a993d113f2d,4e43d75af135421f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:39:17.549 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:17.549 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:17.601 INFO 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:39:17.601 INFO 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:39:17.604 INFO 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:39:17.604 WARN 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:39:17.605 INFO 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:39:17.606 WARN 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:39:17.609 INFO 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:39:17.610 WARN 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:39:17.611 ERROR 26087 --- [ocp-schedules-30,0e4d9a1f2dabafd3,db0c6b2e4ea86f0d] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:39:18.080 INFO 26087 --- [ocp-schedules-25,d96179966c25af02,90de8f3240630e0e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:39:18.083 INFO 26087 --- [ocp-schedules-25,d96179966c25af02,90de8f3240630e0e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:39:18.083 INFO 26087 --- [ocp-schedules-25,d96179966c25af02,90de8f3240630e0e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:39:18.271 INFO 26087 --- [prometheus-minute-schedule-1,fd815725224ddd0e,44cbfb441a308453] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:18.623 WARN 26087 --- [ocp-async-7,84dbe3c8d78f1ad5,1c21b9c8bf1754d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:18.623 INFO 26087 --- [ocp-async-2,7312a5d9d32c78b4,9b055879ebe41b38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:19.095 INFO 26087 --- [ocp-schedules-1,b841049bfd1d0e90,1f77916e9c1be899] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:39:19.096 INFO 26087 --- [ocp-schedules-1,b841049bfd1d0e90,1f77916e9c1be899] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:39:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:39:20.549 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:20.550 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:20.623 WARN 26087 --- [ocp-async-3,4063308be5b97747,734502b2b5a8fe8d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:20.623 INFO 26087 --- [ocp-async-2,ed94716f5a01d2c1,b2bf34571116c578] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:21.271 INFO 26087 --- [prometheus-minute-schedule-1,3bd05d8a74711aa5,f222e7c46592f9fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:22.306 INFO 26087 --- [ocp-schedules-27,7d62a85e2035dc97,f1117c2f3df7794b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:39:22.623 INFO 26087 --- [ocp-async-2,0ed7e4b20bd7107c,58d905b7056a4bfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:22.623 WARN 26087 --- [ocp-async-1,2a635a53065f7f08,29b24d173443d956] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:23.550 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:23.551 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:24.271 INFO 26087 --- [prometheus-minute-schedule-1,ffc2bc314f5a700a,89416ecfc9cd741c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:24.623 WARN 26087 --- [ocp-async-10,b1fb81f1e899bc48,f73a502b04bf9334] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:24.623 INFO 26087 --- [ocp-async-7,ee8587a8e76a6b92,7bae4de66346b3cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:26.442 INFO 26087 --- [ocp-schedules-7,7f149f0cb5ae853e,929f8315dd5eabb9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:39:26.442 INFO 26087 --- [ocp-schedules-9,9a4bb7395a11b7c6,33c0f0a27219ac46] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:39:26.488 INFO 26087 --- [ocp-async-7,ceb0ead932c1dd56,c17f93c4f934e600] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:39:26.551 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:26.551 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:39:26.622816+08:00, lastProcessTime=2025-04-08T21:39:16.623534+08:00 2025-04-08 21:39:26.623 WARN 26087 --- [ocp-async-6,39cc04691f30ffe6,6fabc6bc76accbbf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:26.623 INFO 26087 --- [ocp-async-8,8138f30ceb4a6dfc,5ef0c25a4e94f28d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:26.721 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:39:26.622816+08:00 2025-04-08 21:39:26.731 INFO 26087 --- [ocp-schedules-5,fcc77c3dc9a75086,84878650d00e5ec4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:39:27.271 INFO 26087 --- [prometheus-minute-schedule-1,dada9168936a7363,01c2499af95dee06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:28.623 WARN 26087 --- [ocp-async-9,1d0c1c1250d1d29b,4497bae1377e74b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:28.623 INFO 26087 --- [ocp-async-10,1a5700824f11b7a4,fc7f081235692871] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:29.100 INFO 26087 --- [ocp-schedules-24,f826af8dc311ced5,c0572b70e848037a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:39:29.101 INFO 26087 --- [ocp-schedules-24,f826af8dc311ced5,c0572b70e848037a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:39:29.551 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:29.552 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:30.271 INFO 26087 --- [prometheus-minute-schedule-1,d70413f8a6e60202,9ff5351fb084fdcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:30.624 WARN 26087 --- [ocp-async-7,14747c90c9251fea,3683f761f4736573] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:30.624 INFO 26087 --- [ocp-async-5,46659c6d3a5138bc,272968b6fd8859d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:30.893 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:39:30.893 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:39:30.897 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:39:30.897 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:39:30.897 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:39:30.897 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:39:30.900 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:39:30.900 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:39:30.900 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:39:30.900 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:39:30.922 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:39:30.926 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:39:30.926 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:39:30.926 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:39:30.926 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:39:30.926 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:39:30.931 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:39:31.238 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:39:31.555 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.654454384S 2025-04-08 21:39:31.555 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:39:31.610 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:39:31.610 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:39:31.632 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:39:31.672 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:39:31.672 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:39:31.672 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:39:31.672 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:39:31.672 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:39:31.673 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:39:31.680 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:39:31.780 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] o.s.boot.SpringApplication : Started application in 0.883 seconds (JVM running for 2277.236) 2025-04-08 21:39:31.780 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.883548966S 2025-04-08 21:39:32.039 INFO 26087 --- [ocp-schedules-14,3281fb1c1847bb49,0bd27f086673614a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:39:32.332 INFO 26087 --- [ocp-schedules-10,dcbfd50193ddc474,a0253dd84f3bc82a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:39:32.552 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:32.552 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:32.624 WARN 26087 --- [ocp-async-5,645b4e52a9674a97,f9f36627a0476ddd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:32.624 INFO 26087 --- [ocp-async-1,6cdfe786466c94d7,268a02af3a8b5623] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:33.271 INFO 26087 --- [prometheus-minute-schedule-1,016b78c72603eeb7,100d79b9c52378df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:34.624 WARN 26087 --- [ocp-async-1,e1542e477177c70d,0641ca1a80066cb3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:34.624 INFO 26087 --- [ocp-async-9,8bbd4e9c0e28a614,0a087873da91d553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:39:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:39:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:39:34.912 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:39:34.912 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:39:34.917 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:39:34.917 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 21:39:35.552 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:35.553 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:36.271 INFO 26087 --- [prometheus-minute-schedule-1,42d7756d7ea1cefa,0d5bb1b266fffc79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:36.488 INFO 26087 --- [ocp-async-8,c0b243ceff80d8ef,6666cd965a22b0cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:39:36.504 INFO 26087 --- [ocp-schedules-13,8c442e1c26d58895,7faa61efe6a542c9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:39:36.504 INFO 26087 --- [ocp-schedules-13,8c442e1c26d58895,7faa61efe6a542c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:36.505 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:39:36.624 WARN 26087 --- [ocp-async-7,6c360de1ceea46c6,ac75f3d12d42853d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:36.624 INFO 26087 --- [ocp-async-3,5627daf739b40c23,74002ba9da7918d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:39:36.624281+08:00, lastProcessTime=2025-04-08T21:39:26.622816+08:00 2025-04-08 21:39:36.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:39:36.624281+08:00 2025-04-08 21:39:36.732 INFO 26087 --- [ocp-schedules-18,ef35f7a0b597283d,4848057fe35f5eed] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:39:38.084 INFO 26087 --- [ocp-schedules-1,bf7e7a177cf4cc10,de68edd2732700c9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:39:38.108 INFO 26087 --- [ocp-schedules-1,bf7e7a177cf4cc10,de68edd2732700c9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:39:38.108 INFO 26087 --- [ocp-schedules-1,bf7e7a177cf4cc10,de68edd2732700c9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:39:38.553 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:38.554 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:38.624 WARN 26087 --- [ocp-async-6,8954d77655e2ed2c,e75e75a19190841c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:38.624 INFO 26087 --- [ocp-async-5,d8a3d6a586280067,40302218f6e33096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:39.116 INFO 26087 --- [ocp-schedules-10,ad778a27f65ed4ff,3386600350e39a30] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:39:39.117 INFO 26087 --- [ocp-schedules-10,ad778a27f65ed4ff,3386600350e39a30] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:39:39.271 INFO 26087 --- [prometheus-minute-schedule-1,d1ed78a52be94194,6be5c2ffb11bc6d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:40.001 INFO 26087 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:39:40.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:39:40.624 WARN 26087 --- [ocp-async-8,51f61bfaf0c0166d,e098096d6bbe3904] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:40.624 INFO 26087 --- [ocp-async-2,49b34595712455e2,edf6625c2cf2a6bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:41.554 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:41.554 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:42.271 INFO 26087 --- [prometheus-minute-schedule-1,d2f6ded794bce2b3,abb2b67b846402e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:42.354 INFO 26087 --- [ocp-schedules-12,a885d94797bcf959,a46b5747fafae4b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:39:42.624 WARN 26087 --- [ocp-async-7,0cfdc88d1f2a6019,52314b734ba9346d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:42.624 INFO 26087 --- [ocp-async-2,58e637c94becd020,6ad1dcf317977e46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:44.554 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:44.555 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:44.625 WARN 26087 --- [ocp-async-6,747216d5c7d8eda5,4e93a3b51abf1be5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:44.625 INFO 26087 --- [ocp-async-2,1a75f3519e326d61,1daa5d1854e6ca54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:45.271 INFO 26087 --- [prometheus-minute-schedule-1,f0dc8a52024e2661,6241431cf49c3251] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:46.437 INFO 26087 --- [ocp-schedules-17,b48ee804a2783819,ab48ada8b793c84c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:39:46.438 INFO 26087 --- [ocp-schedules-10,e8d1d6e5efc636b7,70f89cb8824036b2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:39:46.438 INFO 26087 --- [ocp-schedules-10,e8d1d6e5efc636b7,70f89cb8824036b2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:39:46.440 INFO 26087 --- [ocp-schedules-1,bc2db84fe49e0cc0,7cd2f878934bfc12] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:39:46.440 INFO 26087 --- [ocp-schedules-1,bc2db84fe49e0cc0,7cd2f878934bfc12] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:39:46.440 INFO 26087 --- [ocp-schedules-28,65577a464552d810,ec0b58c3925390c5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:39:46.440 INFO 26087 --- [ocp-schedules-28,65577a464552d810,ec0b58c3925390c5] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:39:46.441 INFO 26087 --- [ocp-schedules-23,a4eaa9d286cbb83c,d22cad932592d56c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:39:46.441 INFO 26087 --- [ocp-schedules-18,c48af2013c306809,d7ebd070f0b8596a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:39:46.441 INFO 26087 --- [ocp-schedules-23,a4eaa9d286cbb83c,d22cad932592d56c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:46.441 INFO 26087 --- [ocp-schedules-18,c48af2013c306809,d7ebd070f0b8596a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:39:46.475 INFO 26087 --- [ocp-schedules-11,76fda39955a0c4fa,79e9cd3ef8976c90] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:39:46.488 INFO 26087 --- [ocp-async-4,24efd59f456df552,06116a68ba0c68b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:39:46.505 INFO 26087 --- [ocp-schedules-13,8c442e1c26d58895,7faa61efe6a542c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:39:46.519 INFO 26087 --- [ocp-schedules-15,5d03e9a43cbc5e46,7dd8a326a703d878] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:39:46.519 INFO 26087 --- [ocp-schedules-15,c00880a9fc336cc6,996096067930c34b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:39:46.520 INFO 26087 --- [ocp-schedules-15,c00880a9fc336cc6,996096067930c34b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:39:46.525 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:39:46.525 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:39:46.576 INFO 26087 --- [ocp-schedules-6,6636645178c3bcc6,20b8478d247dc9ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:39:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:39:46.623808+08:00, lastProcessTime=2025-04-08T21:39:36.624281+08:00 2025-04-08 21:39:46.625 WARN 26087 --- [ocp-async-1,45994bff8db77047,f796ffdc97ff8401] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:46.625 INFO 26087 --- [ocp-async-9,f0ba4ae1f52dc3b2,c6e060eb8880923e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:46.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:39:46.623808+08:00 2025-04-08 21:39:46.734 INFO 26087 --- [ocp-schedules-14,5f117760d02e4ddc,ea178376022b5afd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:39:47.555 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:47.556 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:48.271 INFO 26087 --- [prometheus-minute-schedule-1,1c77e825ee438247,3fa41c351e2eba2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:48.625 WARN 26087 --- [ocp-async-9,8314e154b4ad308a,aaeeae34c7b2d904] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:48.625 INFO 26087 --- [ocp-async-3,8a860dd091d169bf,9aa38a428c81d017] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:49.141 INFO 26087 --- [ocp-schedules-5,5262d0d653d55d2c,feea8cbb76f3fd31] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:39:49.142 INFO 26087 --- [ocp-schedules-5,5262d0d653d55d2c,feea8cbb76f3fd31] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:39:50.556 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:50.556 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:50.625 WARN 26087 --- [ocp-async-6,42c794bbdf24eddb,878b0c87b8267f40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:50.625 INFO 26087 --- [ocp-async-4,00ec678bc4ef0771,a494ba0ef615a3d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:51.271 INFO 26087 --- [prometheus-minute-schedule-1,66655957d63d7499,47d4016bb84152d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:52.377 INFO 26087 --- [ocp-schedules-25,5f264b427a199b30,e9805cc7175cae5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:39:52.625 WARN 26087 --- [ocp-async-4,a85fd5963d287657,e7bb153e0e6c2f71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:52.625 INFO 26087 --- [ocp-async-8,4336ebb905dc68fa,28941d4605f60e1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:53.556 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:53.557 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:54.271 INFO 26087 --- [prometheus-minute-schedule-1,3bd017e8db4b07ee,ca8027a0fadfd5df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:54.625 WARN 26087 --- [ocp-async-9,bd9d4e763465f88f,0a57c267f409805b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:54.625 INFO 26087 --- [ocp-async-7,2278475343876276,1e31efd141872e43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:54.918 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,4911de02bb30b979,b57b8178107d3535] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=4911de02bb30b979, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:39:54.919 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-10,4911de02bb30b979,b57b8178107d3535] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=4911de02bb30b979, duration=5 ms] 2025-04-08 21:39:55.706 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,66e1a0b617f5f418,b19c51aaf5611b19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.11.12.102, traceId=66e1a0b617f5f418, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-04-08 21:39:55.707 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-8,66e1a0b617f5f418,b19c51aaf5611b19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.11.12.102, traceId=66e1a0b617f5f418, duration=3 ms] 2025-04-08 21:39:56.442 INFO 26087 --- [ocp-schedules-18,c48af2013c306809,d7ebd070f0b8596a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:39:56.442 INFO 26087 --- [ocp-schedules-23,a4eaa9d286cbb83c,d22cad932592d56c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:39:56.488 INFO 26087 --- [ocp-async-10,7fedbed3bb324e0f,6423e85a8208af25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:39:56.557 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:56.558 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:39:56.625 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:39:56.624527+08:00, lastProcessTime=2025-04-08T21:39:46.623808+08:00 2025-04-08 21:39:56.625 WARN 26087 --- [ocp-async-6,33c459c4358e2374,f80c4900bd981e4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:56.625 INFO 26087 --- [ocp-async-2,4a5e9d70cf35b20f,8e65e5e7633e6b58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:56.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:39:56.624527+08:00 2025-04-08 21:39:56.735 INFO 26087 --- [ocp-schedules-2,7cd6e2efe8cc4b5c,de33aa1512057d41] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:39:57.271 INFO 26087 --- [prometheus-minute-schedule-1,c585ee8ce5886154,0adf2d39aa7299e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119600, interval=60 2025-04-08 21:39:58.081 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,420aba2e377b91a8,e0b9aaf45d41e6f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.11.12.102, traceId=420aba2e377b91a8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-04-08 21:39:58.109 INFO 26087 --- [ocp-schedules-29,b7992e6172335071,9d9788e976769cf7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:39:58.124 INFO 26087 --- [ocp-schedules-29,b7992e6172335071,9d9788e976769cf7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:39:58.124 INFO 26087 --- [ocp-schedules-29,b7992e6172335071,9d9788e976769cf7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:39:58.167 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-2,420aba2e377b91a8,e0b9aaf45d41e6f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.11.12.102, traceId=420aba2e377b91a8, duration=90 ms] 2025-04-08 21:39:58.626 WARN 26087 --- [ocp-async-4,db365ae8d9004ce0,e1b3c6bc59cec707] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:39:58.626 INFO 26087 --- [ocp-async-1,981884a989b37fed,0acebc45caf818bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:39:59.192 INFO 26087 --- [ocp-schedules-1,b031fa1a36bc547b,65ab6d1589b6f1bc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:39:59.193 INFO 26087 --- [ocp-schedules-1,b031fa1a36bc547b,65ab6d1589b6f1bc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:39:59.558 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:39:59.558 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:00.209 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:35:00.0, nextRun=2025-04-08 21:40:00.0, currentDate=Tue Apr 08 21:40:00 CST 2025 2025-04-08 21:40:00.209 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:35:00.0, nextRunTime=2025-04-08 21:40:00.0} 2025-04-08 21:40:00.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:40:00.212 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:40:00.212+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:40:00.221 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:35:00.0, nextRun=2025-04-08 21:40:00.0, currentDate=Tue Apr 08 21:40:00 CST 2025 2025-04-08 21:40:00.221 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:35:00.0, nextRunTime=2025-04-08 21:40:00.0} 2025-04-08 21:40:00.226 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:40:00.228 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:40:00.228 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,cf84000fe317fd90] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:40:00.271 INFO 26087 --- [prometheus-minute-schedule-1,def531ebc134727d,31a7300e996b239a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:00.272 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,22343a12d57fca30] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000165, operation:EXECUTE, state: RUNNING 2025-04-08 21:40:00.287 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:40:00.288 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000165, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000018, task_operation=execute, latest_execution_start_time=2025-04-08T21:40:00.268+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000165}, listMap={}}, executor=192.168.1.61 2025-04-08 21:40:00.292 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.293 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 21:40:00.293 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.293 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.294 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.294 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.294 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.295 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.295 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:40:00.302 INFO 26087 --- [subtask-executor14,8ec1a44aeefd3a62,f2a0ebd93ea7fd8a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000165, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:40:00.626 WARN 26087 --- [ocp-async-7,5e8cd3288e54b565,9e1df1aeeb80ec45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:00.626 INFO 26087 --- [ocp-async-9,b3a95c049dc0fc94,b78b0fdc536cb183] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:01.046 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000018, status:SUCCESS 2025-04-08 21:40:01.046 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000018, status:SUCCESS 2025-04-08 21:40:02.411 INFO 26087 --- [ocp-schedules-28,0893ae3e1bac6f30,f7e29abc87fd4ab5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:40:02.558 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:02.559 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:02.626 WARN 26087 --- [ocp-async-3,989dcf7151e2623e,c07953da7c0eb4d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:02.626 INFO 26087 --- [ocp-async-5,2a916a615a1a66af,cc1119be249e00e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:03.271 INFO 26087 --- [prometheus-minute-schedule-1,cad7918be68ed0b6,133a8ae53916a429] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:04.440 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,bcfc654464aac39b] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:30:05.0, nextRun=2025-04-08 21:40:04.0, currentDate=Tue Apr 08 21:40:04 CST 2025 2025-04-08 21:40:04.441 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,bcfc654464aac39b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-04-08 21:30:05.0, nextRunTime=2025-04-08 21:40:04.0} 2025-04-08 21:40:04.443 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,bcfc654464aac39b] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-04-08 21:40:04.443 INFO 26087 --- [task-coordinator1,8ec1a44aeefd3a62,bcfc654464aac39b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:40:04.443+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:40:04.626 WARN 26087 --- [ocp-async-2,fcb7ef90f0711533,6e4bcb5604a87c43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:04.626 INFO 26087 --- [ocp-async-4,d05993d14c4ff250,215ee88aad9c4ca0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:05.287 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,36d3c0de33d2594f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000166, operation:EXECUTE, state: RUNNING 2025-04-08 21:40:05.306 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:40:05.308 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000166, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000019, task_operation=execute, latest_execution_start_time=2025-04-08T21:40:05.286+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=1000166}, listMap={}}, executor=192.168.1.61 2025-04-08 21:40:05.312 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:40:05.312 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:40:05.313 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.313 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:40:05.314 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.314 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.314 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.315 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:40:05.326 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.327 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.327 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.327 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:40:05.338 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.338 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.338 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.338 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:40:05.349 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.349 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.350 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.350 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:40:05.360 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.360 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.361 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.362 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:40:05.366 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.367 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.367 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.367 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:40:05.371 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.371 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.371 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.371 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:40:05.375 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.375 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.376 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.376 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:40:05.381 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.381 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.381 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.381 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:40:05.385 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.386 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.387 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.387 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:40:05.392 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.392 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.392 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.392 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:40:05.399 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.399 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.400 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.400 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:40:05.410 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.410 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.411 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.411 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:40:05.415 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.415 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.415 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.415 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:40:05.420 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.421 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.421 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.421 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:40:05.426 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.426 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.427 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.427 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:40:05.431 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.431 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.431 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.431 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:40:05.435 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.435 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.435 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.435 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:40:05.438 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.439 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.439 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.439 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:40:05.443 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.443 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.443 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.444 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:40:05.447 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.447 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.448 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.448 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:40:05.451 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.452 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.452 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.452 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:40:05.455 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.455 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.456 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.456 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:40:05.459 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.459 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.459 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.460 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:40:05.463 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.463 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.464 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.464 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:40:05.467 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.467 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.467 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.468 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:40:05.471 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.471 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.471 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.471 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:40:05.475 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.475 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.475 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.475 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:40:05.479 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.479 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.479 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.480 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:40:05.483 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.483 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.483 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.483 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:40:05.486 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.486 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.487 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.487 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:40:05.490 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.490 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.491 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.491 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:40:05.494 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.494 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.495 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.495 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:40:05.498 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.499 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.499 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.499 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:40:05.506 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.506 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.507 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.507 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:40:05.512 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.512 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.512 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.513 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:40:05.517 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.517 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.517 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.518 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:40:05.523 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.523 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.523 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.523 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:40:05.526 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.526 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.526 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.527 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:40:05.530 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.530 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.530 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.531 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:40:05.533 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.533 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.534 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.534 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:40:05.537 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.537 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.537 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.538 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:40:05.545 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.545 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.545 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.545 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:40:05.559 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:05.559 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:05.563 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.564 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.564 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.564 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:40:05.569 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.569 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.569 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.570 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:40:05.574 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.575 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.575 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.575 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:40:05.580 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.581 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.581 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.581 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:40:05.584 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.584 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.585 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.585 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:40:05.589 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.589 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.589 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.589 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:40:05.593 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:40:05.593 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:40:05.594 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.594 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:40:05.595 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.595 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.595 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.595 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:40:05.599 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.599 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:40:05.599 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:05.599 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:40:05.602 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:40:05.653 INFO 26087 --- [subtask-executor15,8ec1a44aeefd3a62,a15da8a1d34ad6cd] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000166, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:40:06.093 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000019, status:SUCCESS 2025-04-08 21:40:06.094 INFO 26087 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000019, status:SUCCESS 2025-04-08 21:40:06.271 INFO 26087 --- [prometheus-minute-schedule-1,82f51074000abebf,5980650708504d26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:06.272 INFO 26087 --- [prometheus-cache-monitor1,8d6b5eda0bb885db,b5b7d812c4f813b2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:40:06.272 INFO 26087 --- [prometheus-cache-monitor1,8d6b5eda0bb885db,b5b7d812c4f813b2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:40:06.488 INFO 26087 --- [ocp-async-5,d8b932d7814b253a,d1156425322cdf07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:40:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:40:06.624252+08:00, lastProcessTime=2025-04-08T21:39:56.624527+08:00 2025-04-08 21:40:06.626 WARN 26087 --- [ocp-async-10,11384cf5e2fff530,5617a818baf67662] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:06.626 INFO 26087 --- [ocp-async-1,0892bddfeb9586f0,618984b1ea283417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:06.737 INFO 26087 --- [ocp-schedules-23,cf2379fa636813e0,d577c308b89baaad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:40:06.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:40:06.624252+08:00 2025-04-08 21:40:06.748 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:36:06.645, end=2025-04-08T21:38:06.703} 2025-04-08 21:40:06.753 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:40:06.753 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:40:08.559 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:08.560 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:08.626 WARN 26087 --- [ocp-async-6,4660a1d1d740ca47,b0fa644217446920] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:08.626 INFO 26087 --- [ocp-async-1,16fb224daad7f242,548ea353508233e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:09.198 INFO 26087 --- [ocp-schedules-12,f1f3d95885984715,61cba5e8a4e307ea] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:40:09.199 INFO 26087 --- [ocp-schedules-12,f1f3d95885984715,61cba5e8a4e307ea] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:40:09.271 INFO 26087 --- [prometheus-minute-schedule-1,d6b25d1df89ce2a9,568afc4a7a782834] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:10.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:40:10.002 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:40:10.522 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:40:10.626 INFO 26087 --- [ocp-async-1,74dac4034772388e,8340e36f3f1b2cce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:10.626 WARN 26087 --- [ocp-async-4,8335deeb13ead040,61b16871ffe41222] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:11.560 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:11.561 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:12.271 INFO 26087 --- [prometheus-minute-schedule-1,15ab49ad0ce7d0ce,0177d667be82949f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:12.440 INFO 26087 --- [ocp-schedules-29,4b7a3a4bd58d3aad,de73bdffd7913e56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:40:12.467 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119540, nodeCount=380, affectRows=380 2025-04-08 21:40:12.469 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119540, nodeCount=377, validCount=377 2025-04-08 21:40:12.627 WARN 26087 --- [ocp-async-1,b5cbb5bea1e15f83,bc3e17525f3e049a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:12.627 INFO 26087 --- [ocp-async-10,31a100ce2581c8e4,ef2645aca297d212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:14.561 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:14.561 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:14.627 WARN 26087 --- [ocp-async-3,68c789ffd0b3be39,f16833fb922de78c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:14.627 INFO 26087 --- [ocp-async-10,7ce1342cfcc850cc,0e75fc0d14d36b10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:15.271 INFO 26087 --- [prometheus-minute-schedule-1,42d7797ee872cb0e,05c9dda4e46ba777] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:16.433 INFO 26087 --- [ocp-schedules-13,a385728109f304e3,ed3dfadeb228511c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:40:16.434 INFO 26087 --- [ocp-schedules-30,ad7d8f9c567d02f0,2f76c273fec636eb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:40:16.437 INFO 26087 --- [ocp-schedules-23,23d4d90cdb44a6ca,2e985d00e2b0f357] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:40:16.437+08:00, tenantSize=0 2025-04-08 21:40:16.437 INFO 26087 --- [ocp-schedules-12,4df1295e706e8292,5893dc173f87a5eb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:40:16.438 INFO 26087 --- [ocp-schedules-6,5505294eb3f73081,6716a73435e51526] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:40:16.438 INFO 26087 --- [ocp-schedules-6,5505294eb3f73081,6716a73435e51526] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:40:16.440 INFO 26087 --- [ocp-schedules-7,4c7e29e10749b56d,3da96c7b57410eed] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:40:16.440 INFO 26087 --- [ocp-schedules-7,4c7e29e10749b56d,3da96c7b57410eed] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:40:16.440 INFO 26087 --- [ocp-schedules-13,a385728109f304e3,ed3dfadeb228511c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:40:16.440 INFO 26087 --- [ocp-schedules-16,5b720a04321de789,7496c97480bd1f4d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:40:16.440 INFO 26087 --- [ocp-schedules-16,5b720a04321de789,7496c97480bd1f4d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:40:16.441 INFO 26087 --- [ocp-schedules-6,1e94917b93877804,79210a38515f7ba0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:40:16.441 INFO 26087 --- [ocp-schedules-1,b802725e589f0ebb,34c90ed362630110] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:40:16.441 INFO 26087 --- [ocp-schedules-6,1e94917b93877804,79210a38515f7ba0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:16.441 INFO 26087 --- [ocp-schedules-1,b802725e589f0ebb,34c90ed362630110] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:40:16.443 INFO 26087 --- [ocp-schedules-30,ad7d8f9c567d02f0,2f76c273fec636eb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:40:16.444 INFO 26087 --- [ocp-schedules-23,23402b36172ca61c,bd9c51368453de89] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:40:16.482 INFO 26087 --- [ocp-schedules-25,78233062b0b7e384,346d5fa0ab51b746] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:40:16.483 INFO 26087 --- [ocp-schedules-25,78233062b0b7e384,346d5fa0ab51b746] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:40:16.488 INFO 26087 --- [ocp-async-4,842fe8a1e4c7f06a,c3e8b40d7cf0613b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:40:16.525 INFO 26087 --- [ocp-schedules-15,c00880a9fc336cc6,996096067930c34b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:40:16.525 INFO 26087 --- [ocp-schedules-15,d2882c704c64637a,acc57b5b5eeb1fd2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:40:16.526 INFO 26087 --- [ocp-schedules-15,d2882c704c64637a,acc57b5b5eeb1fd2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:16.531 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:40:16.531 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:40:16.535 INFO 26087 --- [ocp-schedules-29,8998c2fc4119b6de,abf6d4dbf6142f4c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:40:16.577 INFO 26087 --- [ocp-schedules-9,32a6ab8eee94a54f,ae698e66a354ac9d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:40:16.581 INFO 26087 --- [ocp-schedules-24,b40f28c2bfc7abd7,0702432e62a4b375] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:40:16.581 INFO 26087 --- [ocp-schedules-24,b40f28c2bfc7abd7,0702432e62a4b375] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:40:16.592 INFO 26087 --- [ocp-schedules-22,d7085cdb85778616,480e3cb948503e7e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:40:16.592 INFO 26087 --- [ocp-schedules-22,d7085cdb85778616,480e3cb948503e7e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:40:16.613 INFO 26087 --- [ocp-schedules-7,739b266e646bc926,890581dad01eb005] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:40:16.613+08:00, clusters=[1000003] 2025-04-08 21:40:16.613 INFO 26087 --- [ocp-schedules-7,739b266e646bc926,890581dad01eb005] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:40:16.613 INFO 26087 --- [ocp-schedules-12,7052ff5d7e6fbbbe,c9133383819f9705] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:40:16.617 INFO 26087 --- [ocp-schedules-30,5d703a10ab48dd33,db925e0563452813] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:40:16.618 INFO 26087 --- [ocp-schedules-30,5d703a10ab48dd33,db925e0563452813] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:40:16.618 INFO 26087 --- [ocp-schedules-30,5d703a10ab48dd33,db925e0563452813] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:40:16.619 INFO 26087 --- [ocp-schedules-16,c8caa2f7b95aabfc,2946c80dd34cc245] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:40:16.619+08:00, clusters=[1000003] 2025-04-08 21:40:16.621 INFO 26087 --- [ocp-schedules-16,c8caa2f7b95aabfc,2946c80dd34cc245] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:40:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:40:16.623112+08:00, lastProcessTime=2025-04-08T21:40:06.624252+08:00 2025-04-08 21:40:16.627 WARN 26087 --- [ocp-async-1,a88310f408adf904,8b2c10da167310db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:16.627 INFO 26087 --- [ocp-async-8,ac4e39dee15f528f,c3368f93e10e739f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:16.631 INFO 26087 --- [ocp-schedules-17,be86240136965c70,c3cebbaf1eb48af0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:40:16.631 INFO 26087 --- [ocp-schedules-17,be86240136965c70,c3cebbaf1eb48af0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:40:16.635 INFO 26087 --- [ocp-schedules-17,be86240136965c70,c3cebbaf1eb48af0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:40:16.635 INFO 26087 --- [ocp-schedules-17,be86240136965c70,c3cebbaf1eb48af0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:40:16.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:40:16.623112+08:00 2025-04-08 21:40:16.738 INFO 26087 --- [ocp-schedules-25,ac9f6591052ef0c4,f3b495b618e43aa1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:40:16.809 INFO 26087 --- [ocp-schedules-2,e542c0670e4718c6,15a6bb13d5910813] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:40:16.810 INFO 26087 --- [ocp-schedules-2,e542c0670e4718c6,15a6bb13d5910813] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:16.822 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:40:16.822 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:40:16.828 INFO 26087 --- [ocp-schedules-27,9834aa8660e78194,23c7081673cd64e5] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:40:16.828 INFO 26087 --- [ocp-schedules-27,9834aa8660e78194,23c7081673cd64e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:40:16.831 INFO 26087 --- [ocp-schedules-27,9834aa8660e78194,23c7081673cd64e5] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:40:16.831 INFO 26087 --- [ocp-schedules-27,9834aa8660e78194,23c7081673cd64e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:40:16.833 INFO 26087 --- [ocp-schedules-27,9834aa8660e78194,23c7081673cd64e5] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:40:16.833 INFO 26087 --- [ocp-schedules-27,9834aa8660e78194,23c7081673cd64e5] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:40:16.876 INFO 26087 --- [ocp-schedules-11,08f6ac6947fe491b,846656a14ebc2032] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:40:16.877 INFO 26087 --- [ocp-schedules-11,08f6ac6947fe491b,846656a14ebc2032] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:16.894 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:40:16.895 ERROR 26087 --- [ocp-schedules-11,08f6ac6947fe491b,846656a14ebc2032] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:40:16.895 INFO 26087 --- [ocp-schedules-11,08f6ac6947fe491b,846656a14ebc2032] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:40:16.933 INFO 26087 --- [ocp-schedules-21,a96b36b67efb2112,c6c3e47279a1a6f8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:40:17.561 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:17.562 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:17.614 INFO 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:40:17.614 INFO 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:40:17.616 INFO 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:40:17.617 WARN 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:40:17.618 INFO 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:40:17.618 WARN 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:40:17.621 INFO 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:40:17.622 WARN 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:40:17.623 ERROR 26087 --- [ocp-schedules-13,44b7eecb03771b89,c40f504a6d66ced9] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:40:18.126 INFO 26087 --- [ocp-schedules-27,08f2d7b59f0bace2,398a1ed9100db369] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:40:18.133 INFO 26087 --- [ocp-schedules-27,08f2d7b59f0bace2,398a1ed9100db369] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:40:18.133 INFO 26087 --- [ocp-schedules-27,08f2d7b59f0bace2,398a1ed9100db369] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:40:18.271 INFO 26087 --- [prometheus-minute-schedule-1,291d5e17c806ff93,b0aab14f41bba377] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:18.627 WARN 26087 --- [ocp-async-3,e19938c31bbc7b57,21d4df66d0173b64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:18.627 INFO 26087 --- [ocp-async-8,3eaa2916b358e949,b88d5b47248ab83f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:19.219 INFO 26087 --- [ocp-schedules-23,2e364560cff574e1,1bcf623a0d981062] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:40:19.220 INFO 26087 --- [ocp-schedules-23,2e364560cff574e1,1bcf623a0d981062] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:40:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:40:20.562 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:20.563 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:20.627 WARN 26087 --- [ocp-async-10,b3d2499e768791af,fc8187e632cda462] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:20.627 INFO 26087 --- [ocp-async-4,48b564bcd45942ea,0e9bdc977426523f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:21.271 INFO 26087 --- [prometheus-minute-schedule-1,7de17827aa14ab10,ffa5aad4a18a8543] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:22.467 INFO 26087 --- [ocp-schedules-8,74b9992cd62728cf,ec2ef0d3d4eb5b23] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:40:22.627 WARN 26087 --- [ocp-async-5,b713e92e0d913abe,9448e6881fed979c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:22.627 INFO 26087 --- [ocp-async-1,caa8576ca72b233d,3b4f24aabdb4c72d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:23.563 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:23.563 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:24.271 INFO 26087 --- [prometheus-minute-schedule-1,fbe6fe51bd68d1fc,1d74ceab9b330fd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:24.628 WARN 26087 --- [ocp-async-3,a652a5b15c094734,b358f16e12f834ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:24.628 INFO 26087 --- [ocp-async-7,1c8b48ec7b13e8ad,3e338225c44f4351] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:26.442 INFO 26087 --- [ocp-schedules-1,b802725e589f0ebb,34c90ed362630110] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:40:26.442 INFO 26087 --- [ocp-schedules-6,1e94917b93877804,79210a38515f7ba0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:40:26.489 INFO 26087 --- [ocp-async-8,120ebe79d10c0a6e,8d11a5597360927d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:40:26.563 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:26.564 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:40:26.622781+08:00, lastProcessTime=2025-04-08T21:40:16.623112+08:00 2025-04-08 21:40:26.628 WARN 26087 --- [ocp-async-4,8f2faf5f6a4531b5,04cde8bf5e1cce0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:26.628 INFO 26087 --- [ocp-async-2,b58bc74e01fcdf04,87d0e85cec37fb2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:26.731 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:40:26.622781+08:00 2025-04-08 21:40:26.740 INFO 26087 --- [ocp-schedules-1,bca9017d79f22ead,6e5698055eb32429] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:40:27.271 INFO 26087 --- [prometheus-minute-schedule-1,34998fa8fc77ffd2,a1ca4ddb2aad13cf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:28.628 INFO 26087 --- [ocp-async-2,01966d2f8f9e4bdd,1a0631cc59bfb879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:28.628 WARN 26087 --- [ocp-async-3,14d04f45fcbe7305,51a2eabc31e32203] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:29.243 INFO 26087 --- [ocp-schedules-21,878b86b21cd6b2a2,9060f4d94db8a829] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:40:29.244 INFO 26087 --- [ocp-schedules-21,878b86b21cd6b2a2,9060f4d94db8a829] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:40:29.564 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:29.565 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:30.271 INFO 26087 --- [prometheus-minute-schedule-1,8325e7dd619bf8bf,6db9998df4a6e7ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:30.628 WARN 26087 --- [ocp-async-7,5a1345d0380c0ca9,9302fab4b34adf85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:30.628 INFO 26087 --- [ocp-async-8,d3453caf56aef581,04e4c22daacdfa04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:32.491 INFO 26087 --- [ocp-schedules-23,573bbcba008d7238,3ac8e4823fc4f291] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:40:32.565 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:32.565 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:32.628 WARN 26087 --- [ocp-async-4,21ea73da93b5710d,f205ae896fa30f74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:32.628 INFO 26087 --- [ocp-async-10,bbf2288446f61802,6e4c318abd9dc103] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:33.271 INFO 26087 --- [prometheus-minute-schedule-1,b39f78647e31e553,eb5f72c2a58781f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:34.628 WARN 26087 --- [ocp-async-3,9c136d14ebbd2d99,bf3c7b99d584590d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:34.628 INFO 26087 --- [ocp-async-5,ae05e1b1935cdee0,3ee1e0f0e96d9edd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:35.565 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:35.566 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:36.271 INFO 26087 --- [prometheus-minute-schedule-1,7f8593823f0c77c9,ea5f99136388e8c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:36.489 INFO 26087 --- [ocp-async-2,d77cb587674988f4,bd6d3a3481af419b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:40:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:40:36.624040+08:00, lastProcessTime=2025-04-08T21:40:26.622781+08:00 2025-04-08 21:40:36.628 WARN 26087 --- [ocp-async-6,a135907e8f5e23b1,bb27ce495acc4493] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:36.629 INFO 26087 --- [ocp-async-8,336515ea9b76d3f1,139946e73e196f8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:36.738 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:40:36.624040+08:00 2025-04-08 21:40:36.741 INFO 26087 --- [ocp-schedules-18,1a84e420bb744fa3,50ef51e21ce7647d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:40:38.134 INFO 26087 --- [ocp-schedules-14,f333ebcdf7738e21,16c4878aa3809cbb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:40:38.139 INFO 26087 --- [ocp-schedules-14,f333ebcdf7738e21,16c4878aa3809cbb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:40:38.139 INFO 26087 --- [ocp-schedules-14,f333ebcdf7738e21,16c4878aa3809cbb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:40:38.566 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:38.567 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:38.629 WARN 26087 --- [ocp-async-10,42a385ecd7642a26,36354fb35b9ebf9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:38.629 INFO 26087 --- [ocp-async-9,73870a6d818a46f6,2326f47a47876185] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:39.251 INFO 26087 --- [ocp-schedules-19,fbd95b2b5747cd4f,38b87ea1160b94ce] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:40:39.252 INFO 26087 --- [ocp-schedules-19,fbd95b2b5747cd4f,38b87ea1160b94ce] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:40:39.271 INFO 26087 --- [prometheus-minute-schedule-1,ddb7121cf81a26ea,d7e39b9e81cf501c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:40.001 INFO 26087 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:40:40.001 INFO 26087 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:40:40.629 WARN 26087 --- [ocp-async-1,23e95d9986d9dc07,fd1b2b2ea8f38fb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:40.629 INFO 26087 --- [ocp-async-3,fc8b6bad058df7df,ef2d8a40a3390ccd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:41.567 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:41.567 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:42.271 INFO 26087 --- [prometheus-minute-schedule-1,4111f691b7dbe8ec,63a8741b50b80986] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:42.511 INFO 26087 --- [ocp-schedules-7,284b5711b4972b00,bd47e078666e7bf5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:40:42.629 WARN 26087 --- [ocp-async-6,41505866f23a4b23,c93c92db54b51dd9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:42.629 INFO 26087 --- [ocp-async-3,25c4686850f83c90,f42ce6f0575580cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:44.567 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:44.568 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:44.629 WARN 26087 --- [ocp-async-10,80df1804dbc1e00d,7b21b57cc6d9c317] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:44.629 INFO 26087 --- [ocp-async-4,52d62263ea657810,b2f3929147d97542] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:45.271 INFO 26087 --- [prometheus-minute-schedule-1,d79699d910f86761,a545f28165938710] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:46.437 INFO 26087 --- [ocp-schedules-7,54e157ad53e92122,f46726f58957c27a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:40:46.437 INFO 26087 --- [ocp-schedules-21,85eff7e3c9e2d2c1,b680c90b662d9c35] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:40:46.437 INFO 26087 --- [ocp-schedules-21,85eff7e3c9e2d2c1,b680c90b662d9c35] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:40:46.440 INFO 26087 --- [ocp-schedules-18,dc62eee78fd97406,3074b0d76b5a8752] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:40:46.440 INFO 26087 --- [ocp-schedules-26,f9e5918f9e035216,9b1beae42a823c8c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:40:46.440 INFO 26087 --- [ocp-schedules-18,dc62eee78fd97406,3074b0d76b5a8752] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:40:46.440 INFO 26087 --- [ocp-schedules-26,f9e5918f9e035216,9b1beae42a823c8c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:40:46.440 INFO 26087 --- [ocp-schedules-8,9f2fd23368fcbe93,e2287e80529e5e68] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:40:46.441 INFO 26087 --- [ocp-schedules-6,aab44055fa7e4fbb,9a18ceb10dac34c6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:40:46.441 INFO 26087 --- [ocp-schedules-12,4974f26ed374e7d1,3f8ddf81236b60c2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:40:46.441 INFO 26087 --- [ocp-schedules-6,aab44055fa7e4fbb,9a18ceb10dac34c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:46.441 INFO 26087 --- [ocp-schedules-12,4974f26ed374e7d1,3f8ddf81236b60c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:40:46.489 INFO 26087 --- [ocp-async-8,43082a654573ce43,92b3b10e25a72444] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:40:46.505 INFO 26087 --- [ocp-schedules-5,fbd54fce314b3e0e,51c26e23182e8911] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:40:46.506 INFO 26087 --- [ocp-schedules-5,fbd54fce314b3e0e,51c26e23182e8911] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:46.506 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:40:46.531 INFO 26087 --- [ocp-schedules-15,d2882c704c64637a,acc57b5b5eeb1fd2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:40:46.532 INFO 26087 --- [ocp-schedules-15,1ce8d8013d1d4721,aae36285396a3263] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:40:46.532 INFO 26087 --- [ocp-schedules-15,1ce8d8013d1d4721,aae36285396a3263] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:40:46.537 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:40:46.537 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:40:46.579 INFO 26087 --- [ocp-schedules-17,d02b2ad2151e9400,96febf5e1bcbf232] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:40:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:40:46.623168+08:00, lastProcessTime=2025-04-08T21:40:36.624040+08:00 2025-04-08 21:40:46.629 WARN 26087 --- [ocp-async-2,da3b81bcac7f4c82,69db5e70961a7e64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:46.629 INFO 26087 --- [ocp-async-1,ed730604968ff486,cee8789fe7c485d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:46.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:40:46.623168+08:00 2025-04-08 21:40:46.742 INFO 26087 --- [ocp-schedules-30,00040be758c3f40b,c8ee81c1b7adb89c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:40:47.568 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:47.569 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:48.271 INFO 26087 --- [prometheus-minute-schedule-1,31fc8458f8f7b59a,cffcf991dddfee4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:48.629 WARN 26087 --- [ocp-async-7,6c76ea2aab235193,bb7af94e62f0469c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:48.629 INFO 26087 --- [ocp-async-6,c949f4b450d39056,3ca79b3cefc9f435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:49.296 INFO 26087 --- [ocp-schedules-26,6c455a67854daf97,d0415c35d338f000] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:40:49.297 INFO 26087 --- [ocp-schedules-26,6c455a67854daf97,d0415c35d338f000] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:40:50.569 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:50.569 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:50.630 INFO 26087 --- [ocp-async-10,8dbcc3b0feb90499,b865e026eb571549] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:50.630 WARN 26087 --- [ocp-async-8,d8e9a279fd5895fb,9f1cd7a52d7fd7d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:51.271 INFO 26087 --- [prometheus-minute-schedule-1,aa31f0362ba71182,a2fe1c86567bb0b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:52.536 INFO 26087 --- [ocp-schedules-11,8b08196dc53a7735,b534e61e43a08a0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:40:52.630 WARN 26087 --- [ocp-async-2,3be2dc9712fea44a,a728c4fa1c74cde0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:52.630 INFO 26087 --- [ocp-async-10,f4b3d7803d480879,40da56dcbc7ef896] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:53.569 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:53.570 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:54.271 INFO 26087 --- [prometheus-minute-schedule-1,2e181028a336f99a,d8d67cda945cf2a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:54.630 WARN 26087 --- [ocp-async-1,86cbeffd8978f3a1,7e3a3496c793e143] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:54.630 INFO 26087 --- [ocp-async-10,8baefe3d7c0e65a9,73ee4f2ae85d275f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:56.442 INFO 26087 --- [ocp-schedules-12,4974f26ed374e7d1,3f8ddf81236b60c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:40:56.442 INFO 26087 --- [ocp-schedules-6,aab44055fa7e4fbb,9a18ceb10dac34c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:40:56.489 INFO 26087 --- [ocp-async-7,f7a565aa31cbac64,9719c8518302b798] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:40:56.506 INFO 26087 --- [ocp-schedules-5,fbd54fce314b3e0e,51c26e23182e8911] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:40:56.570 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:56.571 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:40:56.626 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:40:56.625911+08:00, lastProcessTime=2025-04-08T21:40:46.623168+08:00 2025-04-08 21:40:56.630 WARN 26087 --- [ocp-async-6,2888c17fcedcbc2c,2577b81f094b0843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:56.630 INFO 26087 --- [ocp-async-4,d438368b74a84a6b,116eb1a1c342e635] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:56.726 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:40:56.625911+08:00 2025-04-08 21:40:56.744 INFO 26087 --- [ocp-schedules-26,25017acd888c31c6,dcc455ce0492118e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:40:57.271 INFO 26087 --- [prometheus-minute-schedule-1,411d874cff9dfc33,212254d9385469a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119660, interval=60 2025-04-08 21:40:58.140 INFO 26087 --- [ocp-schedules-3,57e512b73368ca3a,b90b9f0f8cf00b3a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:40:58.152 INFO 26087 --- [ocp-schedules-3,57e512b73368ca3a,b90b9f0f8cf00b3a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:40:58.152 INFO 26087 --- [ocp-schedules-3,57e512b73368ca3a,b90b9f0f8cf00b3a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:40:58.630 WARN 26087 --- [ocp-async-5,c7f91f72007b9a00,020feb1ee1c23f2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:40:58.630 INFO 26087 --- [ocp-async-5,e93a0c5f6f9e7901,c886a3106660b48d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:40:59.320 INFO 26087 --- [ocp-schedules-10,99ae86700ecb1d46,c43ca4c7694a226e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:40:59.321 INFO 26087 --- [ocp-schedules-10,99ae86700ecb1d46,c43ca4c7694a226e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:40:59.571 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:40:59.571 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:00.271 INFO 26087 --- [prometheus-minute-schedule-1,aef10c15563e0077,609c9386d0688bf5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:00.630 WARN 26087 --- [ocp-async-3,a57564e248073a70,fa64eb75e38a56c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:00.630 INFO 26087 --- [ocp-async-10,cadb7b9817e01c0a,e25e14b91597f330] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:02.557 INFO 26087 --- [ocp-schedules-25,1980269ee1795416,ef518dc5183aa81d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:41:02.572 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:02.572 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:02.630 WARN 26087 --- [ocp-async-10,b52bb0c1e83f5e07,778178e60d7ac7c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:02.630 INFO 26087 --- [ocp-async-2,5467810941c2df86,c1d5d6e0d60575fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:03.271 INFO 26087 --- [prometheus-minute-schedule-1,d103b2052ea61a23,9c81d14f87c747c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:04.631 WARN 26087 --- [ocp-async-1,1a3cf9b21da2df3e,2a18a01961cb8e74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:04.631 INFO 26087 --- [ocp-async-2,fec8c20e2e551472,5bfe2336af4edd3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:05.572 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:05.573 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:06.271 INFO 26087 --- [prometheus-minute-schedule-1,484a644e5936a5da,173fb3c2a574b581] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:06.272 INFO 26087 --- [prometheus-cache-monitor1,c11ebf2c3d1cf57b,83c1cd1869580186] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:41:06.272 INFO 26087 --- [prometheus-cache-monitor1,c11ebf2c3d1cf57b,83c1cd1869580186] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:41:06.489 INFO 26087 --- [ocp-async-9,30f6ca890261a307,b1b662f230c10edf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:41:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:41:06.623372+08:00, lastProcessTime=2025-04-08T21:40:56.625911+08:00 2025-04-08 21:41:06.631 WARN 26087 --- [ocp-async-7,b8e9e2fcb6f548d9,829715976d7ef3a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:06.631 INFO 26087 --- [ocp-async-5,c04299b6a121669c,7d1e43f00928e1d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:06.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:41:06.623372+08:00 2025-04-08 21:41:06.745 INFO 26087 --- [ocp-schedules-14,f14ef049b413d729,9c4acf4a3ed71f74] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:41:08.573 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:08.574 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:08.631 WARN 26087 --- [ocp-async-5,b2eddd1e3ee2b7c0,b376537ea8a3f247] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:08.631 INFO 26087 --- [ocp-async-6,1293a596d0d26d19,b3dfc49e1dafb669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:09.271 INFO 26087 --- [prometheus-minute-schedule-1,30dc169bc2dbdf62,1ee7fe2dc06c8b7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:09.328 INFO 26087 --- [ocp-schedules-3,428c4d6d5d178e01,ccd5127debeefe6e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:41:09.329 INFO 26087 --- [ocp-schedules-3,428c4d6d5d178e01,ccd5127debeefe6e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:41:10.002 INFO 26087 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:41:10.002 INFO 26087 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:41:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:41:10.631 WARN 26087 --- [ocp-async-4,ab84e0bc4a4e1cf1,3f1efa8452bf01b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:10.631 INFO 26087 --- [ocp-async-2,45018727452a3952,c68c65ff4bdcaec2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:11.574 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:11.574 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:12.271 INFO 26087 --- [prometheus-minute-schedule-1,dcd4d9eaa130808c,0d70de30d23d2f73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:12.479 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119600, nodeCount=380, affectRows=380 2025-04-08 21:41:12.480 INFO 26087 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119600, nodeCount=377, validCount=377 2025-04-08 21:41:12.583 INFO 26087 --- [ocp-schedules-21,ddf25b33e4d09976,b018189a8b5d779b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:41:12.631 WARN 26087 --- [ocp-async-7,e0da61b4b811544d,2fb1718fcd1580d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:12.631 INFO 26087 --- [ocp-async-3,89d4dfa0f3c98925,701d75a6c6b02421] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:14.575 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:14.575 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:14.631 WARN 26087 --- [ocp-async-5,2a8172679b981cf7,2aea5f437bfa7204] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:14.631 INFO 26087 --- [ocp-async-8,9b67314b7325ee23,4bc6b11023371cc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:15.271 INFO 26087 --- [prometheus-minute-schedule-1,b0d785a5a3b037b7,a217ee0daf859270] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:16.433 INFO 26087 --- [ocp-schedules-8,52a6a14b66d91909,877eeda3c0eb8270] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:41:16.434 INFO 26087 --- [ocp-schedules-10,27fbd909a2a0d80e,21c6c5634cf82688] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:41:16.437 INFO 26087 --- [ocp-schedules-23,40dd67e414a1e03b,9c7a01f460b1acda] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:41:16.437 INFO 26087 --- [ocp-schedules-23,40dd67e414a1e03b,9c7a01f460b1acda] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:41:16.437 INFO 26087 --- [ocp-schedules-16,6a03d6510d548d07,c27d94aaa07321f4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:41:16.440 INFO 26087 --- [ocp-schedules-17,b915c9bb56b23594,6eea2880ad285d56] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:41:16.440 INFO 26087 --- [ocp-schedules-24,9efde40480d76c10,4e1ed1ff0b164074] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:41:16.440 INFO 26087 --- [ocp-schedules-24,9efde40480d76c10,4e1ed1ff0b164074] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:41:16.440 INFO 26087 --- [ocp-schedules-17,b915c9bb56b23594,6eea2880ad285d56] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:41:16.440 INFO 26087 --- [ocp-schedules-10,27fbd909a2a0d80e,21c6c5634cf82688] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:41:16.441 INFO 26087 --- [ocp-schedules-3,6fc5efb13dc2d970,16097e3d38ecfcde] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:41:16.441 INFO 26087 --- [ocp-schedules-4,302086c2da7c0e42,9753dadf9804e34a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:41:16.441 INFO 26087 --- [ocp-schedules-3,6fc5efb13dc2d970,16097e3d38ecfcde] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:16.441 INFO 26087 --- [ocp-schedules-4,302086c2da7c0e42,9753dadf9804e34a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:41:16.473 INFO 26087 --- [ocp-schedules-11,cb2b81ba7a1a0d81,14cddc7414a5e410] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2025-04-08 21:41:16.474 INFO 26087 --- [ocp-schedules-11,cb2b81ba7a1a0d81,14cddc7414a5e410] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-04-08 21:41:16.489 INFO 26087 --- [ocp-async-4,6f8bad1fa06e97bc,17d21d90d5794b1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:41:16.505 INFO 26087 --- [ocp-schedules-27,e6154c41d096abcd,c4bd5a594d5df065] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:41:16.505 INFO 26087 --- [ocp-schedules-27,e6154c41d096abcd,c4bd5a594d5df065] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:41:16.537 INFO 26087 --- [ocp-schedules-20,298117d6350c9d2a,bec4563abe365c12] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:41:16.537 INFO 26087 --- [ocp-schedules-15,1ce8d8013d1d4721,aae36285396a3263] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:41:16.537 INFO 26087 --- [ocp-schedules-15,886a3f4facf2957e,3fe6e9fac54373e2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:41:16.538 INFO 26087 --- [ocp-schedules-15,886a3f4facf2957e,3fe6e9fac54373e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:16.543 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:41:16.543 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:41:16.579 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:41:16.579 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:41:16.580 INFO 26087 --- [ocp-schedules-24,1c1a88783edfccf1,9afb42c35853d1b4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:41:16.583 INFO 26087 --- [ocp-schedules-17,a2dd583386064829,03a83dc434816cbe] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:41:16.583 INFO 26087 --- [ocp-schedules-17,a2dd583386064829,03a83dc434816cbe] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:41:16.584 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:41:16.584 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:41:16.584 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:41:16.585 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:41:16.585 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:41:16.585 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:41:16.587 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:41:16.587 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:41:16.587 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:41:16.589 INFO 26087 --- [ocp-schedules-16,a1509e9f50a3791a,9ecc574812939ac3] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:41:16.594 INFO 26087 --- [ocp-schedules-10,be4045782f6799cd,e761644f2463dd0e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:41:16.594 INFO 26087 --- [ocp-schedules-10,be4045782f6799cd,e761644f2463dd0e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:41:16.618 INFO 26087 --- [ocp-schedules-18,0585388eae246d4f,5d335c14ebc81776] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:41:16.618+08:00, clusters=[1000003] 2025-04-08 21:41:16.618 INFO 26087 --- [ocp-schedules-25,fe845a4f1d6e6fb6,89e75dc5e727bb34] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:41:16.618 INFO 26087 --- [ocp-schedules-18,0585388eae246d4f,5d335c14ebc81776] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:41:16.619 INFO 26087 --- [ocp-schedules-28,d888365f84d6117d,58783cab59427a5a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:41:16.620 INFO 26087 --- [ocp-schedules-28,d888365f84d6117d,58783cab59427a5a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:41:16.620 INFO 26087 --- [ocp-schedules-28,d888365f84d6117d,58783cab59427a5a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:41:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:41:16.623705+08:00, lastProcessTime=2025-04-08T21:41:06.623372+08:00 2025-04-08 21:41:16.627 INFO 26087 --- [ocp-schedules-11,2fc96c803c0ae15e,f9e4453cff548b66] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:41:16.627+08:00, clusters=[1000003] 2025-04-08 21:41:16.628 INFO 26087 --- [ocp-schedules-11,2fc96c803c0ae15e,f9e4453cff548b66] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:41:16.631 WARN 26087 --- [ocp-async-6,611ae9eef9f74ed1,8bfe2331d65dadec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:16.631 INFO 26087 --- [ocp-async-2,ad722abb8dd447e7,32628cb599e8b5c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:16.635 INFO 26087 --- [ocp-schedules-13,c1c9c692fd9be246,2b9d915b887e7b35] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:41:16.635 INFO 26087 --- [ocp-schedules-13,c1c9c692fd9be246,2b9d915b887e7b35] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:41:16.640 INFO 26087 --- [ocp-schedules-13,c1c9c692fd9be246,2b9d915b887e7b35] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:41:16.640 INFO 26087 --- [ocp-schedules-13,c1c9c692fd9be246,2b9d915b887e7b35] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:41:16.656 INFO 26087 --- [ocp-schedules-8,52a6a14b66d91909,877eeda3c0eb8270] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:41:16.687 INFO 26087 --- [ocp-schedules-29,40547e07f8ceb2ef,384cf14c7c7fa79f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:41:16.747 INFO 26087 --- [ocp-schedules-5,43d301335df25373,6ad99764602fd0a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:41:16.769 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:41:16.623705+08:00 2025-04-08 21:41:16.790 INFO 26087 --- [ocp-schedules-22,930ef0eb2b23d112,f8735d491e86dfe1] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:41:16.791 INFO 26087 --- [ocp-schedules-22,930ef0eb2b23d112,f8735d491e86dfe1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:16.803 ERROR 26087 --- [ocp-schedules-22,930ef0eb2b23d112,f8735d491e86dfe1] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:41:16.803 INFO 26087 --- [ocp-schedules-22,930ef0eb2b23d112,f8735d491e86dfe1] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:41:16.828 INFO 26087 --- [ocp-schedules-2,e542c0670e4718c6,15a6bb13d5910813] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:41:16.836 INFO 26087 --- [ocp-schedules-27,ebba9ba0a81248ef,c200d83b1b271965] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:41:16.836 INFO 26087 --- [ocp-schedules-27,ebba9ba0a81248ef,c200d83b1b271965] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:41:16.836 INFO 26087 --- [ocp-schedules-27,ebba9ba0a81248ef,c200d83b1b271965] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:41:16.836 INFO 26087 --- [ocp-schedules-27,ebba9ba0a81248ef,c200d83b1b271965] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:41:16.837 INFO 26087 --- [ocp-schedules-27,ebba9ba0a81248ef,c200d83b1b271965] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:41:16.837 INFO 26087 --- [ocp-schedules-27,ebba9ba0a81248ef,c200d83b1b271965] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:41:16.895 INFO 26087 --- [ocp-schedules-12,0b9c9219efb5b128,1f9afcf4c7116956] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:41:16.896 INFO 26087 --- [ocp-schedules-12,0b9c9219efb5b128,1f9afcf4c7116956] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:16.912 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:41:16.913 ERROR 26087 --- [ocp-schedules-12,0b9c9219efb5b128,1f9afcf4c7116956] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:41:16.913 INFO 26087 --- [ocp-schedules-12,0b9c9219efb5b128,1f9afcf4c7116956] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:41:16.957 INFO 26087 --- [ocp-schedules-23,4b5a05e791d8a6c0,e546a3865e2681d5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:41:17.575 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:17.576 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:17.626 INFO 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:41:17.626 INFO 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:41:17.628 INFO 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:41:17.629 WARN 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:41:17.630 INFO 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:41:17.630 WARN 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:41:17.634 INFO 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:41:17.635 WARN 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:41:17.639 ERROR 26087 --- [ocp-schedules-18,a9d66df16b2fcf4a,ee6768c049475763] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:41:18.154 INFO 26087 --- [ocp-schedules-1,bcd3246c08ea1d1d,425f606cca54a1f8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:41:18.169 INFO 26087 --- [ocp-schedules-1,bcd3246c08ea1d1d,425f606cca54a1f8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:41:18.169 INFO 26087 --- [ocp-schedules-1,bcd3246c08ea1d1d,425f606cca54a1f8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:41:18.271 INFO 26087 --- [prometheus-minute-schedule-1,32c6a2c93d8520cc,ec3cd341b62e3bee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:18.632 WARN 26087 --- [ocp-async-5,fbccfdbfd984c885,d0c4e9f89a6a4b11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:18.632 INFO 26087 --- [ocp-async-4,837af36ef62b0f2f,15dce2da9ea720d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:19.333 INFO 26087 --- [ocp-schedules-19,21b83345e4b32263,ec23c97e9a14ca23] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:41:19.335 INFO 26087 --- [ocp-schedules-19,21b83345e4b32263,ec23c97e9a14ca23] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:41:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:41:20.576 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:20.576 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:20.632 INFO 26087 --- [ocp-async-9,e01421e9546a28a7,df8c18ee099db26e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:20.632 WARN 26087 --- [ocp-async-6,3aa9d31993bc6629,ecf5552e4c72c00e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:21.271 INFO 26087 --- [prometheus-minute-schedule-1,2f024cac258bfa23,28587d42f2cbe6fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:22.603 INFO 26087 --- [ocp-schedules-30,06a93355525d3c42,4b5b17747f578923] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:41:22.632 WARN 26087 --- [ocp-async-7,4dfb90a575492872,63f7b227d49dc7bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:22.632 INFO 26087 --- [ocp-async-3,1ee7606975c412bd,e3660ae510514516] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:23.577 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:23.577 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:24.271 INFO 26087 --- [prometheus-minute-schedule-1,adf4fe47d9598f19,f12f059e29754eec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:24.632 WARN 26087 --- [ocp-async-2,f97c10b7d93422f1,eeb1af2a26167689] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:24.632 INFO 26087 --- [ocp-async-1,06f4f8ddbf964b6c,ac844d83b2b1f156] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:26.442 INFO 26087 --- [ocp-schedules-4,302086c2da7c0e42,9753dadf9804e34a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:41:26.442 INFO 26087 --- [ocp-schedules-3,6fc5efb13dc2d970,16097e3d38ecfcde] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:41:26.489 INFO 26087 --- [ocp-async-8,ad4e18037b1a938b,21d6e00688ddbb28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:41:26.577 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:26.578 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:41:26.624017+08:00, lastProcessTime=2025-04-08T21:41:16.623705+08:00 2025-04-08 21:41:26.632 INFO 26087 --- [ocp-async-9,9cece1d7ed996a0a,970bdc1fb9c0c6c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:26.632 WARN 26087 --- [ocp-async-7,0168b971d38a0bfc,5780cd1823978c0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:26.734 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:41:26.624017+08:00 2025-04-08 21:41:26.748 INFO 26087 --- [ocp-schedules-10,37885f1497228492,a4ee3e76bd2c722b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:41:27.271 INFO 26087 --- [prometheus-minute-schedule-1,423a6090dad028e9,2843cd48d3351167] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:28.632 WARN 26087 --- [ocp-async-3,c18a912577d2a02e,1bd4bfd309e3b9bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:28.632 INFO 26087 --- [ocp-async-10,5e436c626e9fffb3,f0af2d62e17dcec0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:29.383 INFO 26087 --- [ocp-schedules-14,ac49d21eb10c6741,376f28503f66f9a2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:41:29.384 INFO 26087 --- [ocp-schedules-14,ac49d21eb10c6741,376f28503f66f9a2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:41:29.578 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:29.578 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:30.271 INFO 26087 --- [prometheus-minute-schedule-1,cd70549857ab63bc,8228b88e75008e79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:30.633 WARN 26087 --- [ocp-async-5,69fb40c7cfaaa4af,cf3039c7ee0f876f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:30.633 INFO 26087 --- [ocp-async-1,1f27bc3b280e40e7,11601ccd42f86b2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:32.579 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:32.579 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:32.628 INFO 26087 --- [ocp-schedules-6,0c859a0522f8c5f2,4cabded3855f9179] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:41:32.633 WARN 26087 --- [ocp-async-6,d27aa8fdb18ac27a,de94b000b0daed54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:32.633 INFO 26087 --- [ocp-async-7,b9b3ad97fc24dcae,3d532e03174c5e4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:33.271 INFO 26087 --- [prometheus-minute-schedule-1,433d9c4e94439a1f,6e46de1c5af963f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:34.633 WARN 26087 --- [ocp-async-7,04465e0fdae32f60,15aee285195e1c8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:34.633 INFO 26087 --- [ocp-async-3,c7fb4271f71c1e0e,736595c5a052489b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:34.894 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:41:34.899 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:41:34.900 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 21:41:35.579 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:35.580 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:36.271 INFO 26087 --- [prometheus-minute-schedule-1,fb20e159ed6e90e5,a2a7bd39828c1f65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:36.490 INFO 26087 --- [ocp-async-2,e63441e2fdc288da,810dbc377426baa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:41:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:41:36.623809+08:00, lastProcessTime=2025-04-08T21:41:26.624017+08:00 2025-04-08 21:41:36.633 WARN 26087 --- [ocp-async-5,8521425aadfea1eb,0d4aaa2b6f11e010] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:36.633 INFO 26087 --- [ocp-async-1,e6a6527b1a5757aa,c2910ff2840fd5cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:36.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:41:36.623809+08:00 2025-04-08 21:41:36.750 INFO 26087 --- [ocp-schedules-26,486b009416c3ad4d,de48a8cf09565e9b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:41:38.170 INFO 26087 --- [ocp-schedules-19,2bc57f1c4396d8c2,99e494764f774598] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:41:38.175 INFO 26087 --- [ocp-schedules-19,2bc57f1c4396d8c2,99e494764f774598] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:41:38.175 INFO 26087 --- [ocp-schedules-19,2bc57f1c4396d8c2,99e494764f774598] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:41:38.580 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:38.581 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:38.633 WARN 26087 --- [ocp-async-6,b30846209a9730a3,2d11b6b6a765e80a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:38.633 INFO 26087 --- [ocp-async-6,cc38fd05c2104985,b3c5732fa80d4904] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:39.271 INFO 26087 --- [prometheus-minute-schedule-1,f1f4392ea048ed51,c4acf7b2cd19fd05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:39.400 INFO 26087 --- [ocp-schedules-29,9119cd23d03c18f1,25ccb5403f6d22ce] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:41:39.401 INFO 26087 --- [ocp-schedules-29,9119cd23d03c18f1,25ccb5403f6d22ce] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:41:40.001 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:41:40.001 INFO 26087 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:41:40.633 WARN 26087 --- [ocp-async-7,fded0729152fffec,db6e2f577bb9e807] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:40.633 INFO 26087 --- [ocp-async-2,499035ee62135ae6,628ba8a9ff98cc13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:41.581 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:41.581 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:42.271 INFO 26087 --- [prometheus-minute-schedule-1,644f2ebe0d8f692c,f45e25ec3214d442] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:42.633 WARN 26087 --- [ocp-async-10,2b1a035d013b4013,ac6060a5db06fbb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:42.633 INFO 26087 --- [ocp-async-8,7afaf93c7003d460,ef96abdd87c9b3e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:42.650 INFO 26087 --- [ocp-schedules-27,e35b96732d054bee,f0578ee51eb1e924] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:41:44.581 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:44.582 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:44.634 WARN 26087 --- [ocp-async-4,7d2211086b30c090,def6e672542ff7cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:44.634 INFO 26087 --- [ocp-async-9,f160f9e6418a1f74,bc45834d3801de4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:45.271 INFO 26087 --- [prometheus-minute-schedule-1,0742eaf5760b2764,88c412e2166f3a5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:46.437 INFO 26087 --- [ocp-schedules-4,9aba6c7a80e2f3f9,79d0f05e4b2116d0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:41:46.438 INFO 26087 --- [ocp-schedules-27,771eaee0d4af673f,1ae3f2a41a89c993] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:41:46.438 INFO 26087 --- [ocp-schedules-27,771eaee0d4af673f,1ae3f2a41a89c993] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:41:46.439 INFO 26087 --- [ocp-schedules-12,972ab34a6c02debe,30b91e0d4d01e9ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:41:46.440 INFO 26087 --- [ocp-schedules-12,972ab34a6c02debe,30b91e0d4d01e9ad] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:41:46.440 INFO 26087 --- [ocp-schedules-8,df3345ba0b4d7492,205c871ffefe0216] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:41:46.440 INFO 26087 --- [ocp-schedules-8,df3345ba0b4d7492,205c871ffefe0216] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:41:46.441 INFO 26087 --- [ocp-schedules-18,cd5aff8695bc86d8,1a6a659d0aaf21cf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:41:46.441 INFO 26087 --- [ocp-schedules-25,8c1f54cd512df73c,33076098a51f98be] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:41:46.441 INFO 26087 --- [ocp-schedules-18,cd5aff8695bc86d8,1a6a659d0aaf21cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:46.441 INFO 26087 --- [ocp-schedules-25,8c1f54cd512df73c,33076098a51f98be] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:41:46.490 INFO 26087 --- [ocp-schedules-1,c3f83334932f7e30,23e4d0a73d90ed49] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:41:46.490 INFO 26087 --- [ocp-async-8,d69ae5dafaea669b,ad5bf058159a4dd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:41:46.543 INFO 26087 --- [ocp-schedules-15,886a3f4facf2957e,3fe6e9fac54373e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:41:46.544 INFO 26087 --- [ocp-schedules-15,4880da6267953785,fc248105421193f9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:41:46.544 INFO 26087 --- [ocp-schedules-15,4880da6267953785,fc248105421193f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:46.549 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:41:46.549 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:41:46.582 INFO 26087 --- [ocp-schedules-13,63a2d05c0cde137f,0348439cd570e7a7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:41:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:41:46.623720+08:00, lastProcessTime=2025-04-08T21:41:36.623809+08:00 2025-04-08 21:41:46.634 WARN 26087 --- [ocp-async-2,b5fcbcad6f259ecb,fce83a174882f002] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:46.634 INFO 26087 --- [ocp-async-3,09d38701755970be,169d78180aea5de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:46.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:41:46.623720+08:00 2025-04-08 21:41:46.751 INFO 26087 --- [ocp-schedules-7,02045f5212a186b1,b9de8683b078aca6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:41:47.582 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:47.583 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:48.271 INFO 26087 --- [prometheus-minute-schedule-1,6a1dae9a6b344c96,686ea0354a757e82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:48.634 INFO 26087 --- [ocp-async-5,0ee4e38119052cd3,766968898735caae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:48.634 WARN 26087 --- [ocp-async-1,44603dca8005c8f8,5398c5c36b77174b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:49.406 INFO 26087 --- [ocp-schedules-14,e9e2a85036255ea6,ad322c665ad921ff] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:41:49.408 INFO 26087 --- [ocp-schedules-14,e9e2a85036255ea6,ad322c665ad921ff] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:41:50.583 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:50.583 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:50.634 WARN 26087 --- [ocp-async-4,52c12e92c9b53aa1,5d98281d4f6606f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:50.634 INFO 26087 --- [ocp-async-8,ef6d824167d81bbe,37e09a2e12606df7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:51.271 INFO 26087 --- [prometheus-minute-schedule-1,86c46f9d8ede3997,425d40b4a43e57ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:52.634 INFO 26087 --- [ocp-async-8,b8b9d2ba27969281,3c1530e1971077f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:52.634 WARN 26087 --- [ocp-async-7,212585492dcc3b41,f61a8f019ea1ef5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:52.677 INFO 26087 --- [ocp-schedules-8,2b4d7fb83ea7d848,1a74275ec3864005] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:41:53.583 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:53.584 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:54.271 INFO 26087 --- [prometheus-minute-schedule-1,bf127e3962f02119,4427f950500935ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:54.634 WARN 26087 --- [ocp-async-1,d39ae68c37652b44,8d64bebb71424f1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:54.634 INFO 26087 --- [ocp-async-3,2c5e21e35dc4dc6c,92a4dd0863033521] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:56.442 INFO 26087 --- [ocp-schedules-25,8c1f54cd512df73c,33076098a51f98be] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:41:56.442 INFO 26087 --- [ocp-schedules-18,cd5aff8695bc86d8,1a6a659d0aaf21cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:41:56.490 INFO 26087 --- [ocp-async-8,b6f54fb2246ed393,66d3c05a30ca7142] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:41:56.506 INFO 26087 --- [ocp-schedules-4,cb1343fb0e2d2501,481f920ff55fefa9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:41:56.507 INFO 26087 --- [ocp-schedules-4,cb1343fb0e2d2501,481f920ff55fefa9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:41:56.507 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:41:56.584 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:56.585 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:41:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:41:56.623070+08:00, lastProcessTime=2025-04-08T21:41:46.623720+08:00 2025-04-08 21:41:56.634 WARN 26087 --- [ocp-async-9,c6272087339ded05,0cc54fa58e871f8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:56.634 INFO 26087 --- [ocp-async-3,fa5d80c8e0fbbca1,d06131066caac969] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:56.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:41:56.623070+08:00 2025-04-08 21:41:56.753 INFO 26087 --- [ocp-schedules-21,667bd7e91dfcd834,86a9614c9d50b1b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:41:57.271 INFO 26087 --- [prometheus-minute-schedule-1,d5b0296f07713ba9,8ec17799ecb23d4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119720, interval=60 2025-04-08 21:41:58.176 INFO 26087 --- [ocp-schedules-19,207479b6d854de3a,f19ee13765622e3f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:41:58.184 INFO 26087 --- [ocp-schedules-19,207479b6d854de3a,f19ee13765622e3f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:41:58.184 INFO 26087 --- [ocp-schedules-19,207479b6d854de3a,f19ee13765622e3f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:41:58.634 WARN 26087 --- [ocp-async-2,7075fe1038cbe896,54b5b2062fd629c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:41:58.634 INFO 26087 --- [ocp-async-7,f3ab6ad312a92a91,7717630bafc70986] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:41:59.412 INFO 26087 --- [ocp-schedules-1,dfea5c77a5ef626e,fb059e3d7f281572] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:41:59.413 INFO 26087 --- [ocp-schedules-1,dfea5c77a5ef626e,fb059e3d7f281572] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:41:59.585 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:41:59.585 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:00.271 INFO 26087 --- [prometheus-minute-schedule-1,4d9937df2ed17f63,b5bfd7e269d27fbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:00.635 WARN 26087 --- [ocp-async-8,40559b92cd7116dc,e58253cd3c068dfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:00.635 INFO 26087 --- [ocp-async-7,23cfef1ded8e35f5,603ab84cc704e58b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:02.585 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:02.586 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:02.635 WARN 26087 --- [ocp-async-9,2ff25523d47ce0a7,28ac1add58b4d4a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:02.635 INFO 26087 --- [ocp-async-6,8b24225d75c1c2e1,051adb713db26a58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:02.699 INFO 26087 --- [ocp-schedules-26,f5ef3fa69a572b3a,7040d1b0090562be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:42:03.271 INFO 26087 --- [prometheus-minute-schedule-1,3038cb7305a1622e,7ac50de63ecba91c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:04.635 WARN 26087 --- [ocp-async-6,0c5d674c18dfd693,8e3e8c947c2578b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:04.635 INFO 26087 --- [ocp-async-1,a28d225a911c0979,b1c3fd693b265148] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:05.586 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:05.587 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:06.271 INFO 26087 --- [prometheus-minute-schedule-1,e7b3b66bb7c7b9d8,a3c692352091ce39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:06.272 INFO 26087 --- [prometheus-cache-monitor1,d483b5b72f51b1e6,3554067a7f06886e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:42:06.272 INFO 26087 --- [prometheus-cache-monitor1,d483b5b72f51b1e6,3554067a7f06886e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:42:06.490 INFO 26087 --- [ocp-async-1,8deea21ee1463236,950ee67fa14d7f75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:42:06.507 INFO 26087 --- [ocp-schedules-4,cb1343fb0e2d2501,481f920ff55fefa9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:42:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:42:06.624117+08:00, lastProcessTime=2025-04-08T21:41:56.623070+08:00 2025-04-08 21:42:06.635 WARN 26087 --- [ocp-async-8,389536d1fcd2154e,118bce4917698edb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:06.635 INFO 26087 --- [ocp-async-10,1335931e30e167f8,b85b2954f05ec39f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:06.751 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:42:06.624117+08:00 2025-04-08 21:42:06.754 INFO 26087 --- [ocp-schedules-21,9b13fd6c7c0f172b,6a1f90ee804ffbdc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:42:06.807 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:38:06.703, end=2025-04-08T21:40:06.806} 2025-04-08 21:42:06.809 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:42:06.809 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:42:08.587 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:08.587 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:08.635 WARN 26087 --- [ocp-async-7,2102f1ab2edba799,46e34435dd3bcdce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:08.635 INFO 26087 --- [ocp-async-4,9d75e91178140f5b,a8bf30461aa36337] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:09.271 INFO 26087 --- [prometheus-minute-schedule-1,630119309ff3b8b0,cf788c948a9188f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:09.443 INFO 26087 --- [ocp-schedules-30,19781e35648414f2,97d7db2b85b0474a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:42:09.445 INFO 26087 --- [ocp-schedules-30,19781e35648414f2,97d7db2b85b0474a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:42:10.002 INFO 26087 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:42:10.002 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:42:10.515 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:42:10.635 WARN 26087 --- [ocp-async-4,79a62c1a29a5b559,94057076256ceefc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:10.635 INFO 26087 --- [ocp-async-1,b97de9adae2e9f83,bb0fe6fffa68ea76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:11.587 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:11.588 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:12.271 INFO 26087 --- [prometheus-minute-schedule-1,24388a8ac8e9e4fc,46d48646dcb2a1bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:12.461 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119660, nodeCount=53, affectRows=53 2025-04-08 21:42:12.462 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119660, nodeCount=52, validCount=52 2025-04-08 21:42:12.467 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119660, nodeCount=91, affectRows=91 2025-04-08 21:42:12.467 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119660, nodeCount=113, affectRows=113 2025-04-08 21:42:12.468 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119660, nodeCount=90, validCount=90 2025-04-08 21:42:12.468 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119660, nodeCount=113, validCount=113 2025-04-08 21:42:12.476 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119660, nodeCount=123, affectRows=123 2025-04-08 21:42:12.477 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119660, nodeCount=122, validCount=122 2025-04-08 21:42:12.636 WARN 26087 --- [ocp-async-9,57c43358dc449164,ea94a9bb69379355] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:12.636 INFO 26087 --- [ocp-async-1,7cadf651acb7ddf1,9ef5c086433253bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:12.722 INFO 26087 --- [ocp-schedules-12,ee51d673217309e2,097530eed0c217d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:42:14.588 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:14.589 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:14.636 WARN 26087 --- [ocp-async-6,ae1a004ed5f1a749,d284d2aadb0fda9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:14.636 INFO 26087 --- [ocp-async-5,783b4f4abfcbd495,b9a92f11ae64b572] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:15.271 INFO 26087 --- [prometheus-minute-schedule-1,c848445a9dcdc060,39f0a5e23779ed91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:16.433 INFO 26087 --- [ocp-schedules-2,7ae56f94dda86a18,5b5d0cdad47913fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:42:16.434 INFO 26087 --- [ocp-schedules-20,fa2d8e1469be9fcf,02da5b51f2ad40fe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:42:16.437 INFO 26087 --- [ocp-schedules-10,3cc77fa72a043c69,d7fa50e225451f11] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:42:16.437 INFO 26087 --- [ocp-schedules-10,3cc77fa72a043c69,d7fa50e225451f11] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:42:16.437 INFO 26087 --- [ocp-schedules-14,4a56c5e8f7af9a50,772c406e3a910573] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:42:16.439 INFO 26087 --- [ocp-schedules-2,7ae56f94dda86a18,5b5d0cdad47913fb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:42:16.440 INFO 26087 --- [ocp-schedules-26,f2dea9a8e32d0e52,db142a5dc1e6f647] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:42:16.440 INFO 26087 --- [ocp-schedules-26,f2dea9a8e32d0e52,db142a5dc1e6f647] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:42:16.440 INFO 26087 --- [ocp-schedules-18,926dc2e46c014ab9,4d1bb7b54c003f25] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:42:16.440 INFO 26087 --- [ocp-schedules-18,926dc2e46c014ab9,4d1bb7b54c003f25] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:42:16.441 INFO 26087 --- [ocp-schedules-11,356dfb2c2994e402,555d8ba90e3e3271] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:42:16.441 INFO 26087 --- [ocp-schedules-3,77b93760e3de6442,85f658fcce3e92a4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:42:16.441 INFO 26087 --- [ocp-schedules-20,fa2d8e1469be9fcf,02da5b51f2ad40fe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:42:16.441 INFO 26087 --- [ocp-schedules-11,356dfb2c2994e402,555d8ba90e3e3271] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:16.441 INFO 26087 --- [ocp-schedules-3,77b93760e3de6442,85f658fcce3e92a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:42:16.450 INFO 26087 --- [ocp-schedules-4,5455c10eff94b534,4fccd251c65a1c82] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:42:16.490 INFO 26087 --- [ocp-async-4,4dcc384b06bd38ef,54b557aa5d915be1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:42:16.538 INFO 26087 --- [ocp-schedules-24,f4f616544333a673,95497fb51f6adfd6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:42:16.549 INFO 26087 --- [ocp-schedules-15,4880da6267953785,fc248105421193f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:42:16.549 INFO 26087 --- [ocp-schedules-15,c88193d287b5c248,71e8edcfe36b8e2f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:42:16.550 INFO 26087 --- [ocp-schedules-15,c88193d287b5c248,71e8edcfe36b8e2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:16.555 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:42:16.555 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:42:16.583 INFO 26087 --- [ocp-schedules-1,5412a88ce8071ef9,641feb3cd76c9ee2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:42:16.584 INFO 26087 --- [ocp-schedules-7,f02edbcf8fb6badc,b17df13000589448] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:42:16.584 INFO 26087 --- [ocp-schedules-7,f02edbcf8fb6badc,b17df13000589448] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:42:16.595 INFO 26087 --- [ocp-schedules-10,10134d3aee0bc188,db3ed3e493015ae9] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:42:16.595 INFO 26087 --- [ocp-schedules-10,10134d3aee0bc188,db3ed3e493015ae9] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:42:16.620 INFO 26087 --- [ocp-schedules-26,a94cc86400ff09a9,beda8f0eeb777dd9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:42:16.622 INFO 26087 --- [ocp-schedules-26,a94cc86400ff09a9,beda8f0eeb777dd9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:42:16.622 INFO 26087 --- [ocp-schedules-26,a94cc86400ff09a9,beda8f0eeb777dd9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:42:16.623 INFO 26087 --- [ocp-schedules-14,ce3b7c55621af365,7f768d87e4484a1a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:42:16.623+08:00, clusters=[1000003] 2025-04-08 21:42:16.623 INFO 26087 --- [ocp-schedules-21,a23ecb0f26644017,0e0376ed45a53ebf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:42:16.623 INFO 26087 --- [ocp-schedules-14,ce3b7c55621af365,7f768d87e4484a1a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:42:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:42:16.623758+08:00, lastProcessTime=2025-04-08T21:42:06.624117+08:00 2025-04-08 21:42:16.633 INFO 26087 --- [ocp-schedules-18,418dead8c0405a50,1c46d38917e36a61] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:42:16.633+08:00, clusters=[1000003] 2025-04-08 21:42:16.635 INFO 26087 --- [ocp-schedules-18,418dead8c0405a50,1c46d38917e36a61] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:42:16.636 WARN 26087 --- [ocp-async-8,33aeebd4a78beeb4,f2aa213a18b72dba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:16.636 INFO 26087 --- [ocp-async-9,cacb9d9e9003931a,a529dd96449df583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:16.640 INFO 26087 --- [ocp-schedules-4,992b91db284dc142,3696bd033661b1c3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:42:16.640 INFO 26087 --- [ocp-schedules-4,992b91db284dc142,3696bd033661b1c3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:42:16.645 INFO 26087 --- [ocp-schedules-4,992b91db284dc142,3696bd033661b1c3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:42:16.645 INFO 26087 --- [ocp-schedules-4,992b91db284dc142,3696bd033661b1c3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:42:16.720 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:42:16.623758+08:00 2025-04-08 21:42:16.755 INFO 26087 --- [ocp-schedules-25,6d05c36e73317488,30217cc0814e248c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:42:16.828 INFO 26087 --- [ocp-schedules-8,60792aa4a0fb5035,6e54079ff2e5bcc2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:42:16.829 INFO 26087 --- [ocp-schedules-8,60792aa4a0fb5035,6e54079ff2e5bcc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:16.837 INFO 26087 --- [ocp-schedules-28,f6132478d416995a,6b434bb148aa66cf] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:42:16.837 INFO 26087 --- [ocp-schedules-28,f6132478d416995a,6b434bb148aa66cf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:42:16.837 INFO 26087 --- [ocp-schedules-28,f6132478d416995a,6b434bb148aa66cf] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:42:16.837 INFO 26087 --- [ocp-schedules-28,f6132478d416995a,6b434bb148aa66cf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:42:16.837 INFO 26087 --- [ocp-schedules-28,f6132478d416995a,6b434bb148aa66cf] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:42:16.838 INFO 26087 --- [ocp-schedules-28,f6132478d416995a,6b434bb148aa66cf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:42:16.839 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:42:16.839 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:42:16.913 INFO 26087 --- [ocp-schedules-29,f6e0b1813d82b370,b6ea306d25b608c8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:42:16.914 INFO 26087 --- [ocp-schedules-29,f6e0b1813d82b370,b6ea306d25b608c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:16.930 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:42:16.931 ERROR 26087 --- [ocp-schedules-29,f6e0b1813d82b370,b6ea306d25b608c8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:42:16.931 INFO 26087 --- [ocp-schedules-29,f6e0b1813d82b370,b6ea306d25b608c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:42:17.009 INFO 26087 --- [ocp-schedules-27,e0fcbd2ac1d3abe9,c1c5c7b3dfaef99d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:42:17.589 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:17.589 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:17.643 INFO 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:42:17.643 INFO 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:42:17.645 INFO 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:42:17.646 WARN 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:42:17.647 INFO 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:42:17.648 WARN 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:42:17.651 INFO 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:42:17.652 WARN 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:42:17.652 ERROR 26087 --- [ocp-schedules-16,c46e175209375474,0b37301b7b617402] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:42:18.186 INFO 26087 --- [ocp-schedules-25,3513d8ef3058b4eb,033e43b8a44fd713] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:42:18.194 INFO 26087 --- [ocp-schedules-25,3513d8ef3058b4eb,033e43b8a44fd713] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:42:18.194 INFO 26087 --- [ocp-schedules-25,3513d8ef3058b4eb,033e43b8a44fd713] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:42:18.271 INFO 26087 --- [prometheus-minute-schedule-1,bb0f19d5facd91a5,504f4dc9441e3b70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:18.636 INFO 26087 --- [ocp-async-6,de7f257eac35cc15,63d10c829a8d1115] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:18.636 WARN 26087 --- [ocp-async-9,61cddb3b4c013604,6a8b3fe275e09037] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:19.462 INFO 26087 --- [ocp-schedules-29,9f3951498ad2a1d5,09af34c2d3b7fa3c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:42:19.463 INFO 26087 --- [ocp-schedules-29,9f3951498ad2a1d5,09af34c2d3b7fa3c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:42:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:42:20.589 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:20.590 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:20.636 WARN 26087 --- [ocp-async-5,4f13a8412ff41195,a2b62a7b47be9ad4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:20.636 INFO 26087 --- [ocp-async-6,66b419502f2ae0c1,3c6da23f37537d12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:21.271 INFO 26087 --- [prometheus-minute-schedule-1,f845192d4c7d821f,06b733751a53b29b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:22.636 INFO 26087 --- [ocp-async-2,d33a3a9688feb5a5,70defc7c1c9afbc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:22.636 WARN 26087 --- [ocp-async-3,8d808abe5291bbc3,0b868a131da4fd93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:22.744 INFO 26087 --- [ocp-schedules-23,dc3565e50c4de1bf,6bc2f60106f33185] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:42:23.590 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:23.591 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:24.271 INFO 26087 --- [prometheus-minute-schedule-1,55d62ee768f67c1b,d49616aed7dae242] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:24.636 INFO 26087 --- [ocp-async-10,f54886c088b2eec9,6a0702519ebdf753] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:24.636 WARN 26087 --- [ocp-async-9,f398b8ab4b917a45,39e683106560f657] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:26.442 INFO 26087 --- [ocp-schedules-3,77b93760e3de6442,85f658fcce3e92a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:42:26.442 INFO 26087 --- [ocp-schedules-11,356dfb2c2994e402,555d8ba90e3e3271] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:42:26.490 INFO 26087 --- [ocp-async-10,d4b3cf1bd159e1b7,1588b32057f7df6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:42:26.591 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:26.591 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:42:26.623848+08:00, lastProcessTime=2025-04-08T21:42:16.623758+08:00 2025-04-08 21:42:26.637 WARN 26087 --- [ocp-async-4,073ae05a19a56310,7bf919ace3eee65d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:26.637 INFO 26087 --- [ocp-async-3,9b28fcfc4ebe7252,21433520ef0f7a57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:26.704 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:42:26.623848+08:00 2025-04-08 21:42:26.757 INFO 26087 --- [ocp-schedules-1,c849c3d6fb67a7b2,b13288d5009f6f67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:42:27.271 INFO 26087 --- [prometheus-minute-schedule-1,de56ca4f04d492ea,1c57fd35558ea1e1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:28.637 INFO 26087 --- [ocp-async-8,c64af5eb487484a2,9260d95171c399a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:28.637 WARN 26087 --- [ocp-async-9,6afdba2bbaf2fb3a,6ff10ad98b38467e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:29.505 INFO 26087 --- [ocp-schedules-26,1e2ecd65df6b9b3d,9ba243da39bbb32d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:42:29.507 INFO 26087 --- [ocp-schedules-26,1e2ecd65df6b9b3d,9ba243da39bbb32d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:42:29.591 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:29.592 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:30.271 INFO 26087 --- [prometheus-minute-schedule-1,67b061931451f788,c425f019dd889017] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:30.637 WARN 26087 --- [ocp-async-10,556f8377ac04bd8b,8068df1c3e0f89f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:30.637 INFO 26087 --- [ocp-async-2,0bd792e629fa6ed5,09ab991bbebb0fd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:32.040 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:42:32.040 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:42:32.043 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:42:32.043 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:42:32.043 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:42:32.043 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:42:32.046 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:42:32.046 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:42:32.046 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:42:32.046 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:42:32.074 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:42:32.079 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:42:32.079 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:42:32.079 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:42:32.079 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:42:32.079 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:42:32.083 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:42:32.388 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:42:32.592 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:32.592 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:32.637 INFO 26087 --- [ocp-async-4,08e9ed3479291bac,87ae303f534e1c78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:32.637 WARN 26087 --- [ocp-async-6,c0e4c9154a6641de,fb758e3a0fbcdabe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:32.718 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.672395383S 2025-04-08 21:42:32.719 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:42:32.770 INFO 26087 --- [ocp-schedules-23,184ab69ac6012542,cd2392f3f70105cc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:42:32.777 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:42:32.777 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:42:32.794 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:42:32.829 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:42:32.829 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:42:32.829 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:42:32.829 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:42:32.829 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:42:32.830 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:42:32.836 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:42:32.934 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] o.s.boot.SpringApplication : Started application in 0.891 seconds (JVM running for 2458.39) 2025-04-08 21:42:32.934 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.891222528S 2025-04-08 21:42:33.183 INFO 26087 --- [ocp-schedules-14,63c964685611d600,e6b283ab052b90ea] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:42:33.271 INFO 26087 --- [prometheus-minute-schedule-1,846e7ff56bbac482,7c427bb390b1db58] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:34.637 WARN 26087 --- [ocp-async-1,44807bc2afcbfb7a,fc173a88d08a190b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:34.637 INFO 26087 --- [ocp-async-3,90218c4223d2dafd,3b761908da4b0e0a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:35.593 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:35.593 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:36.271 INFO 26087 --- [prometheus-minute-schedule-1,1627efcad3560b59,2ad3b4609561daf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:36.490 INFO 26087 --- [ocp-async-9,524f3fbf85f1905b,613b114f6c6a93a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:42:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:42:36.624021+08:00, lastProcessTime=2025-04-08T21:42:26.623848+08:00 2025-04-08 21:42:36.637 WARN 26087 --- [ocp-async-2,7821e11758b33e6f,c8d6f1cd0221d7e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:36.637 INFO 26087 --- [ocp-async-7,cfaf6c041c55882b,3eb7b516db372d13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:36.731 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:42:36.624021+08:00 2025-04-08 21:42:36.758 INFO 26087 --- [ocp-schedules-25,8b5c737919489ebb,5d72717f43a52958] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:42:38.196 INFO 26087 --- [ocp-schedules-25,3fae6cc52d4dcee9,25002330de6b597e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:42:38.210 INFO 26087 --- [ocp-schedules-25,3fae6cc52d4dcee9,25002330de6b597e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:42:38.210 INFO 26087 --- [ocp-schedules-25,3fae6cc52d4dcee9,25002330de6b597e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:42:38.593 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:38.594 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:38.638 INFO 26087 --- [ocp-async-5,c4b142c8fd708a3d,937d6d19aa384ae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:38.638 WARN 26087 --- [ocp-async-4,a500609f75fdab15,db5dfa3c31430526] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:39.271 INFO 26087 --- [prometheus-minute-schedule-1,6561f6d0d86ced89,80c36caa19849828] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:39.540 INFO 26087 --- [ocp-schedules-26,361bbffa12e8c8f1,d3a10d41eefb5361] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:42:39.542 INFO 26087 --- [ocp-schedules-26,361bbffa12e8c8f1,d3a10d41eefb5361] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:42:40.001 INFO 26087 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:42:40.002 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:42:40.638 WARN 26087 --- [ocp-async-5,e306fcd7dcffd2ac,6a8bb3d7dc561557] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:40.638 INFO 26087 --- [ocp-async-8,8b7eeed92c0597d7,d2a8cd37ef6441ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:41.594 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:41.595 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:42.271 INFO 26087 --- [prometheus-minute-schedule-1,d15516d682143f41,396a050343c9d91d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:42.638 WARN 26087 --- [ocp-async-3,db361f137f0f8ff4,aa2655b7889062d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:42.638 INFO 26087 --- [ocp-async-10,67c2e66efc44dea0,a0be615fe6f60c49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:42.790 INFO 26087 --- [ocp-schedules-10,21b5e587de145bb5,968a955f369ab29b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:42:44.595 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:44.595 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:44.638 WARN 26087 --- [ocp-async-7,f6522c1d3c5ca540,06cd74f3d1e6519a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:44.638 INFO 26087 --- [ocp-async-4,6de98c230fe65488,6006d981c5613ae6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:45.271 INFO 26087 --- [prometheus-minute-schedule-1,a86bcfd8e2a2bc77,c6ad113ca1877c77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:46.437 INFO 26087 --- [ocp-schedules-24,3da03b9bfce0c396,8a6435e7e126c0ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:42:46.438 INFO 26087 --- [ocp-schedules-22,6bbbe3dc907993ff,cc70577f53562ac4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:42:46.438 INFO 26087 --- [ocp-schedules-22,6bbbe3dc907993ff,cc70577f53562ac4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:42:46.440 INFO 26087 --- [ocp-schedules-17,08f0153541e3b3d0,619fe4d8db976af3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:42:46.440 INFO 26087 --- [ocp-schedules-17,08f0153541e3b3d0,619fe4d8db976af3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:42:46.440 INFO 26087 --- [ocp-schedules-20,2ab7bb99df7bd6af,9cd9bd7797d4368b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:42:46.440 INFO 26087 --- [ocp-schedules-20,2ab7bb99df7bd6af,9cd9bd7797d4368b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:42:46.441 INFO 26087 --- [ocp-schedules-16,83209bb9919b9d28,fd9b849ccf8acf17] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:42:46.441 INFO 26087 --- [ocp-schedules-29,cddce3c664076e65,02beec1edd6191fb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:42:46.441 INFO 26087 --- [ocp-schedules-16,83209bb9919b9d28,fd9b849ccf8acf17] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:46.442 INFO 26087 --- [ocp-schedules-29,cddce3c664076e65,02beec1edd6191fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:42:46.443 INFO 26087 --- [ocp-schedules-23,2f445d5ebdcd815f,ba25ca7ca002ddab] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:42:46.491 INFO 26087 --- [ocp-async-6,fc60cee973097c1d,d4dc117308f4da8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:42:46.555 INFO 26087 --- [ocp-schedules-15,c88193d287b5c248,71e8edcfe36b8e2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:42:46.556 INFO 26087 --- [ocp-schedules-15,1e9f4a3982b3c065,a4888cd8aed843e1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:42:46.556 INFO 26087 --- [ocp-schedules-15,1e9f4a3982b3c065,a4888cd8aed843e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:42:46.562 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:42:46.562 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:42:46.584 INFO 26087 --- [ocp-schedules-27,f06fa2d925da070b,e41e38fe9cf965d6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:42:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:42:46.623884+08:00, lastProcessTime=2025-04-08T21:42:36.624021+08:00 2025-04-08 21:42:46.638 WARN 26087 --- [ocp-async-8,bb0150b43a35a967,c1d2db16bdd42600] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:46.638 INFO 26087 --- [ocp-async-9,f3df06c64c9a4d1b,54f1ae33e1dda386] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:46.706 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:42:46.623884+08:00 2025-04-08 21:42:46.760 INFO 26087 --- [ocp-schedules-26,135e88f6bd643a6d,d68a5bbbd42cabe8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:42:47.595 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:47.596 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:48.271 INFO 26087 --- [prometheus-minute-schedule-1,2fc409037886ba50,b95de95938f6a680] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:48.638 WARN 26087 --- [ocp-async-3,5f28a23e68980ee0,80625d2ee36d3523] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:48.638 INFO 26087 --- [ocp-async-10,8d15d7782a288a22,41383b1d748c880e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:49.564 INFO 26087 --- [ocp-schedules-4,1ae49c94dc0a89ac,3e56038d207aa1b3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:42:49.565 INFO 26087 --- [ocp-schedules-4,1ae49c94dc0a89ac,3e56038d207aa1b3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:42:50.596 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:50.597 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:50.638 WARN 26087 --- [ocp-async-6,242cb40308ed4329,0260c592bd2523a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:50.638 INFO 26087 --- [ocp-async-4,e043f54576330fcc,404477794cfa61bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:51.271 INFO 26087 --- [prometheus-minute-schedule-1,40d5785f97c25008,3afd3ef5979d98a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:52.638 WARN 26087 --- [ocp-async-8,e801ad9adb4fdb09,cf59054e3b13bcbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:52.638 INFO 26087 --- [ocp-async-1,7fd1addd684e70a9,653010d7942de082] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:52.814 INFO 26087 --- [ocp-schedules-3,60a8522cb500b930,4161b13034525752] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:42:53.597 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:53.598 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:54.271 INFO 26087 --- [prometheus-minute-schedule-1,7b3198dd5543e61e,09912f8d46ac8304] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:54.639 WARN 26087 --- [ocp-async-3,eb6df33db2a666e6,accfc7820939ee2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:54.639 INFO 26087 --- [ocp-async-9,3e3af13aeafc7ddf,9f14376d97d233b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:56.442 INFO 26087 --- [ocp-schedules-29,cddce3c664076e65,02beec1edd6191fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:42:56.442 INFO 26087 --- [ocp-schedules-16,83209bb9919b9d28,fd9b849ccf8acf17] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:42:56.491 INFO 26087 --- [ocp-async-1,b7765f59d8c37665,ad62b94d4b6cec49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:42:56.598 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:56.598 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:42:56.622126+08:00, lastProcessTime=2025-04-08T21:42:46.623884+08:00 2025-04-08 21:42:56.639 WARN 26087 --- [ocp-async-6,c2f6e2a254847b85,c9c3da8ecad812be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:56.639 INFO 26087 --- [ocp-async-10,6bb93b389f933816,52fd9cedbdf16fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:56.745 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:42:56.622126+08:00 2025-04-08 21:42:56.761 INFO 26087 --- [ocp-schedules-16,3572680a8d245045,15c8aee94d6f3dea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:42:57.271 INFO 26087 --- [prometheus-minute-schedule-1,570ab0a2bc83b398,8113e6c73ae42f99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119780, interval=60 2025-04-08 21:42:58.212 INFO 26087 --- [ocp-schedules-7,2f8300769f181091,503b92a266076cbd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:42:58.243 INFO 26087 --- [ocp-schedules-7,2f8300769f181091,503b92a266076cbd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:42:58.243 INFO 26087 --- [ocp-schedules-7,2f8300769f181091,503b92a266076cbd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:42:58.639 WARN 26087 --- [ocp-async-8,9951d32ef2105273,79dcaa314b5790c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:42:58.639 INFO 26087 --- [ocp-async-5,e1b6a2ab6ed4b89d,95fd7a02e4ad11b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:42:59.599 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:42:59.599 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:42:59.614 INFO 26087 --- [ocp-schedules-27,7cee669f6f9b0c98,62dde337335d8d8e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:42:59.615 INFO 26087 --- [ocp-schedules-27,7cee669f6f9b0c98,62dde337335d8d8e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:43:00.271 INFO 26087 --- [prometheus-minute-schedule-1,03e56bf78cd3d0c4,b7164588c6ba6042] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:00.639 WARN 26087 --- [ocp-async-5,1fb2915af45e844a,7b75f93192df34f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:00.639 INFO 26087 --- [ocp-async-1,89c39c14acf5685a,f6b1835b0f2c6165] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:02.599 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:02.600 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:02.639 WARN 26087 --- [ocp-async-6,68c65e805fdae267,3547d654ad04e8c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:02.639 INFO 26087 --- [ocp-async-1,894dabc8fd1640cb,8568c21ad7c238ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:02.835 INFO 26087 --- [ocp-schedules-5,12cd6987220bee3f,50c2f5b0d6b3dbc7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:43:03.271 INFO 26087 --- [prometheus-minute-schedule-1,00a326eb0c4ae2d0,430baba170931b50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:04.639 WARN 26087 --- [ocp-async-1,42e0a7d947e68cd9,a78d644a8861a63c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:04.639 INFO 26087 --- [ocp-async-3,aa20b9b3f8f7cfa2,7a5fe03a7031e078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:05.600 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:05.600 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:06.271 INFO 26087 --- [prometheus-minute-schedule-1,c06dce7c0280d872,dae4574d4a2b3364] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:06.272 INFO 26087 --- [prometheus-cache-monitor1,0cbf28babcc7321a,74ebe205261077e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:43:06.272 INFO 26087 --- [prometheus-cache-monitor1,0cbf28babcc7321a,74ebe205261077e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:43:06.491 INFO 26087 --- [ocp-async-8,91a20d8828aacfcd,ca6323bc1f68d3dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:43:06.507 INFO 26087 --- [ocp-schedules-5,7fc81ce3919cbdda,64b2847f7150df84] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:43:06.508 INFO 26087 --- [ocp-schedules-5,7fc81ce3919cbdda,64b2847f7150df84] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:06.508 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:43:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:43:06.623219+08:00, lastProcessTime=2025-04-08T21:42:56.622126+08:00 2025-04-08 21:43:06.640 WARN 26087 --- [ocp-async-3,78d7667e7682cb48,8251e47eb00db745] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:06.640 INFO 26087 --- [ocp-async-2,ffb7306d7dd8609c,7c10290d23489d01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:06.716 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:43:06.623219+08:00 2025-04-08 21:43:06.762 INFO 26087 --- [ocp-schedules-30,1d7ac296b80aec13,7afb960cfcb21e8b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:43:08.601 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:08.601 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:08.640 WARN 26087 --- [ocp-async-2,8bc74a2ebe0ee55d,09facc8b991eebf9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:08.640 INFO 26087 --- [ocp-async-4,f7df98addc7b0ff7,1dcc911fc730d7af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:09.271 INFO 26087 --- [prometheus-minute-schedule-1,18ef10a55baaca6c,41ee29a80e0e2698] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:09.619 INFO 26087 --- [ocp-schedules-23,6086c2368728595d,4bc6b9faac0eb6eb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:43:09.620 INFO 26087 --- [ocp-schedules-23,6086c2368728595d,4bc6b9faac0eb6eb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:43:10.001 INFO 26087 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:43:10.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:43:10.520 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:43:10.640 WARN 26087 --- [ocp-async-8,10b03a551da6e816,1d284275855307bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:10.640 INFO 26087 --- [ocp-async-10,32a22d41c421a61b,44147eabe4c3942b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:11.601 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:11.602 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:12.271 INFO 26087 --- [prometheus-minute-schedule-1,b6f6ba4b3b9f6ba6,770730506e105aaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:12.521 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119720, nodeCount=380, affectRows=380 2025-04-08 21:43:12.522 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119720, nodeCount=377, validCount=377 2025-04-08 21:43:12.640 INFO 26087 --- [ocp-async-3,dac86ce8ad725238,0d9c981d8c6c90db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:12.640 WARN 26087 --- [ocp-async-9,b0d921f0cfda7e5b,dd3ddfd4ca46b31d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:12.860 INFO 26087 --- [ocp-schedules-17,8b51faa92b1561f7,be24afff58e05b4b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:43:14.602 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:14.603 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:14.640 WARN 26087 --- [ocp-async-2,f1cd6a1d9fa724e8,ecdd5fec3b0de039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:14.640 INFO 26087 --- [ocp-async-7,41a2b6952e6ef762,0d26290e78671cc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:15.271 INFO 26087 --- [prometheus-minute-schedule-1,2b41bda33b2351e2,fd0522420fb41d2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:16.433 INFO 26087 --- [ocp-schedules-3,836d30fbdc4a444d,cad9a27132b2573a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:43:16.434 INFO 26087 --- [ocp-schedules-27,7a41129f7ee99de8,14576230aae97ed4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:43:16.437 INFO 26087 --- [ocp-schedules-6,322d0586706c39a6,a06571bb90ad9d53] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:43:16.437 INFO 26087 --- [ocp-schedules-6,322d0586706c39a6,a06571bb90ad9d53] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:43:16.437 INFO 26087 --- [ocp-schedules-24,ec03aeb8e40aabda,160b604dd7ce03e7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:43:16.439 INFO 26087 --- [ocp-schedules-11,c4c9856b56852eb6,82239b480bdb48f3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:43:16.439 INFO 26087 --- [ocp-schedules-11,c4c9856b56852eb6,82239b480bdb48f3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:43:16.440 INFO 26087 --- [ocp-schedules-9,5c6fdbeae6f8f5e0,99bdd8f157cca011] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:43:16.440 INFO 26087 --- [ocp-schedules-9,5c6fdbeae6f8f5e0,99bdd8f157cca011] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:43:16.441 INFO 26087 --- [ocp-schedules-7,75aa5bcef55fab13,cc41b6fe56621e3d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:43:16.441 INFO 26087 --- [ocp-schedules-27,7a41129f7ee99de8,14576230aae97ed4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:43:16.441 INFO 26087 --- [ocp-schedules-27,bb0c22bb1a1ad66e,a0203037c236a909] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:43:16.441 INFO 26087 --- [ocp-schedules-7,75aa5bcef55fab13,cc41b6fe56621e3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.441 INFO 26087 --- [ocp-schedules-27,bb0c22bb1a1ad66e,a0203037c236a909] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:43:16.453 INFO 26087 --- [ocp-schedules-23,12616eb4b2d4d67a,0ea4a1d2f26f6f4d] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:43:16.453 INFO 26087 --- [ocp-schedules-23,12616eb4b2d4d67a,0ea4a1d2f26f6f4d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:43:16.456 INFO 26087 --- [ocp-schedules-3,836d30fbdc4a444d,cad9a27132b2573a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:43:16.459 INFO 26087 --- [ocp-schedules-30,0cb740d9ef0cbd91,a14e68413a6cbe25] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:43:16.465 INFO 26087 --- [ocp-schedules-29,5c80af7f0e3e9e82,05bb0eef3291d774] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:43:16.465 INFO 26087 --- [ocp-schedules-29,5c80af7f0e3e9e82,05bb0eef3291d774] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.470 INFO 26087 --- [pool-overview-container-4,314e307ff7d8888d,5464dc394629eda3] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:43:16.470 INFO 26087 --- [pool-overview-container-7,7765ec32be88579e,b546f4411f586e31] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:43:16.475 INFO 26087 --- [pool-11-thread-3,7765ec32be88579e,bcfaeda614091d2b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:43:16.475 INFO 26087 --- [pool-11-thread-2,7765ec32be88579e,d8dabd618fdcdbae] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:43:16.475 INFO 26087 --- [pool-11-thread-4,7765ec32be88579e,2e21e972a2f874df] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:43:16.475 INFO 26087 --- [pool-11-thread-1,7765ec32be88579e,89700fbc83c01c23] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:43:16.477 INFO 26087 --- [ForkJoinPool-7-worker-8,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:43:16.477 INFO 26087 --- [ForkJoinPool-7-worker-8,7765ec32be88579e,c2dc0addf943116d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 21:43:16.477 INFO 26087 --- [ForkJoinPool-7-worker-8,7765ec32be88579e,c2dc0addf943116d] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000003, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:43:16.477 INFO 26087 --- [ForkJoinPool-7-worker-8,7765ec32be88579e,c2dc0addf943116d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 21:43:16.478 INFO 26087 --- [pool-11-thread-2,7765ec32be88579e,d8dabd618fdcdbae] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:43:16.478 INFO 26087 --- [pool-11-thread-4,7765ec32be88579e,2e21e972a2f874df] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:43:16.478 INFO 26087 --- [pool-11-thread-4,7765ec32be88579e,2e21e972a2f874df] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:43:16.483 INFO 26087 --- [ocp-schedules-10,fcb19d904cea5d5b,e629aef257e2a4d4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:43:16.483 INFO 26087 --- [ocp-schedules-10,fcb19d904cea5d5b,e629aef257e2a4d4] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:43:16.490 INFO 26087 --- [pool-11-thread-1,7765ec32be88579e,89700fbc83c01c23] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:43:16.491 INFO 26087 --- [ocp-async-8,6ee845970487bac6,924fb6c12efaa2a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:43:16.492 INFO 26087 --- [ForkJoinPool-7-worker-8,7765ec32be88579e,c2dc0addf943116d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:43:16.492 INFO 26087 --- [ForkJoinPool-7-worker-8,7765ec32be88579e,896750aef02e0a2b] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000003, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:43:16.493 INFO 26087 --- [pool-11-thread-3,7765ec32be88579e,bcfaeda614091d2b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:43:16.493 INFO 26087 --- [pool-11-thread-1,7765ec32be88579e,89700fbc83c01c23] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:43:16.493 INFO 26087 --- [pool-11-thread-1,7765ec32be88579e,89700fbc83c01c23] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:43:16.505 INFO 26087 --- [ocp-schedules-1,18e8b9b73bde6020,d46d71b5db5a79f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:43:16.505 INFO 26087 --- [ocp-schedules-1,18e8b9b73bde6020,d46d71b5db5a79f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:43:16.508 INFO 26087 --- [ocp-schedules-5,7fc81ce3919cbdda,64b2847f7150df84] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:43:16.539 INFO 26087 --- [ocp-schedules-16,a2397e4ad746dd72,ace6a298620ee32e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:43:16.562 INFO 26087 --- [ocp-schedules-15,1e9f4a3982b3c065,a4888cd8aed843e1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:43:16.562 INFO 26087 --- [ocp-schedules-15,e5c397d355085f96,1ccae33956291bde] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:43:16.563 INFO 26087 --- [ocp-schedules-15,e5c397d355085f96,1ccae33956291bde] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.568 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:43:16.568 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:43:16.585 INFO 26087 --- [ocp-schedules-3,597e7478e8923f08,ee806b697827e70a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:43:16.586 INFO 26087 --- [ocp-schedules-28,e0c1e3ec877c8478,de61842b795460ef] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:43:16.586 INFO 26087 --- [ocp-schedules-28,e0c1e3ec877c8478,de61842b795460ef] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:43:16.590 INFO 26087 --- [ocp-schedules-9,6e053cabb1962422,883d5cede9e48ce0] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-04-08 21:43:16.590 INFO 26087 --- [ocp-schedules-9,6e053cabb1962422,883d5cede9e48ce0] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-04-08 21:43:16.590 INFO 26087 --- [ocp-schedules-9,6e053cabb1962422,883d5cede9e48ce0] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=428 2025-04-08 21:43:16.597 INFO 26087 --- [ocp-schedules-4,08947100f8717994,32309960f223bb86] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:43:16.597 INFO 26087 --- [ocp-schedules-4,08947100f8717994,32309960f223bb86] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:43:16.602 INFO 26087 --- [ocp-schedules-9,6e053cabb1962422,883d5cede9e48ce0] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=428 2025-04-08 21:43:16.622 INFO 26087 --- [ocp-schedules-13,c8b82d7dc0d841e6,fddcc5dd22f64a30] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:43:16.624 INFO 26087 --- [ocp-schedules-13,c8b82d7dc0d841e6,fddcc5dd22f64a30] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:43:16.624 INFO 26087 --- [ocp-schedules-13,c8b82d7dc0d841e6,fddcc5dd22f64a30] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:43:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:43:16.623518+08:00, lastProcessTime=2025-04-08T21:43:06.623219+08:00 2025-04-08 21:43:16.628 INFO 26087 --- [ocp-schedules-25,4746727ee7285fb1,45b3aa10220cf6d4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:43:16.628 INFO 26087 --- [ocp-schedules-17,1da87d988270586d,c07ac1ef5fb13ee7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:43:16.628+08:00, clusters=[1000003] 2025-04-08 21:43:16.629 INFO 26087 --- [ocp-schedules-17,1da87d988270586d,c07ac1ef5fb13ee7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:43:16.639 INFO 26087 --- [ocp-schedules-22,0078300efb2ad215,3dd50a4cb96db924] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:43:16.639+08:00, clusters=[1000003] 2025-04-08 21:43:16.640 INFO 26087 --- [ocp-schedules-22,0078300efb2ad215,3dd50a4cb96db924] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:43:16.641 WARN 26087 --- [ocp-async-9,c5454e1b23291416,13ad8fd754647c01] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:16.641 INFO 26087 --- [ocp-async-5,be04336efe0a8972,053d3921f887c368] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:16.645 INFO 26087 --- [ocp-schedules-18,8626144803799121,85fecca53606117e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:43:16.645 INFO 26087 --- [ocp-schedules-18,8626144803799121,85fecca53606117e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:43:16.649 INFO 26087 --- [ocp-schedules-18,8626144803799121,85fecca53606117e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:43:16.650 INFO 26087 --- [ocp-schedules-18,8626144803799121,85fecca53606117e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:43:16.690 INFO 26087 --- [ocp-schedules-20,4788bfd762bc0eca,c93cbb73fddca8f6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-04-08 21:43:16.691 INFO 26087 --- [ocp-schedules-20,4788bfd762bc0eca,c93cbb73fddca8f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.700 ERROR 26087 --- [ocp-schedules-20,4788bfd762bc0eca,c93cbb73fddca8f6] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncClusterSystemEvent(ClusterSchedules.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createClusterOperator(AbstractObOperatorFactory.java:239) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getClusterOperator(ClusterSystemEventServiceImpl.java:229) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.getRootServiceEvents(ClusterSystemEventServiceImpl.java:192) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:162) at com.oceanbase.ocp.obops.internal.cluster.ClusterSystemEventServiceImpl.syncSystemEvents(ClusterSystemEventServiceImpl.java:113) at com.oceanbase.ocp.service.task.business.cluster.SyncSystemEventTask.processCluster(SyncSystemEventTask.java:21) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:43:16.700 INFO 26087 --- [ocp-schedules-20,4788bfd762bc0eca,c93cbb73fddca8f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=1, successCount=0 2025-04-08 21:43:16.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:43:16.623518+08:00 2025-04-08 21:43:16.756 INFO 26087 --- [ocp-schedules-19,1c1da14a489536d5,9c45e2eec95a666a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:43:16.756 INFO 26087 --- [ocp-schedules-19,1c1da14a489536d5,9c45e2eec95a666a] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:43:16.761 INFO 26087 --- [pool-30-thread-1,1c1da14a489536d5,dedd7e37e3e3b63c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:43:16.764 INFO 26087 --- [ocp-schedules-6,de19191b4c4d8368,1a738b19075b44d2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:43:16.766 WARN 26087 --- [ocp-schedules-19,1c1da14a489536d5,9c45e2eec95a666a] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$14662e73.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:43:16.766 INFO 26087 --- [ocp-schedules-19,1c1da14a489536d5,9c45e2eec95a666a] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:43:16.803 INFO 26087 --- [ocp-schedules-5,4943c66d5d1a916d,0de9457c01cd89df] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:43:16.804 INFO 26087 --- [ocp-schedules-5,4943c66d5d1a916d,0de9457c01cd89df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.815 ERROR 26087 --- [ocp-schedules-5,4943c66d5d1a916d,0de9457c01cd89df] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:43:16.815 INFO 26087 --- [ocp-schedules-5,4943c66d5d1a916d,0de9457c01cd89df] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:43:16.838 INFO 26087 --- [ocp-schedules-16,73e4354476cbf470,d741a8382d290603] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:43:16.838 INFO 26087 --- [ocp-schedules-16,73e4354476cbf470,d741a8382d290603] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:43:16.841 INFO 26087 --- [ocp-schedules-16,73e4354476cbf470,d741a8382d290603] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:43:16.841 INFO 26087 --- [ocp-schedules-16,73e4354476cbf470,d741a8382d290603] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:43:16.841 INFO 26087 --- [ocp-schedules-16,73e4354476cbf470,d741a8382d290603] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:43:16.841 INFO 26087 --- [ocp-schedules-16,73e4354476cbf470,d741a8382d290603] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:43:16.845 INFO 26087 --- [ocp-schedules-8,60792aa4a0fb5035,6e54079ff2e5bcc2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:43:16.932 INFO 26087 --- [ocp-schedules-23,dd3ccf143bbfe523,ce81b109fb71b4ea] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:43:16.933 INFO 26087 --- [ocp-schedules-23,dd3ccf143bbfe523,ce81b109fb71b4ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:16.948 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:43:16.949 ERROR 26087 --- [ocp-schedules-23,dd3ccf143bbfe523,ce81b109fb71b4ea] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:43:16.949 INFO 26087 --- [ocp-schedules-23,dd3ccf143bbfe523,ce81b109fb71b4ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:43:17.027 INFO 26087 --- [ocp-schedules-30,df93d24ac548a5f2,110038b48dcf8368] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:43:17.092 INFO 26087 --- [ocp-schedules-28,7513d5e0563f5c3d,2952a76fe12ff163] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:43:17.098 INFO 26087 --- [ocp-schedules-28,7513d5e0563f5c3d,2952a76fe12ff163] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:43:17.099 WARN 26087 --- [ocp-schedules-28,7513d5e0563f5c3d,2952a76fe12ff163] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:43:17.099 INFO 26087 --- [ocp-schedules-28,7513d5e0563f5c3d,2952a76fe12ff163] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:43:17.100 WARN 26087 --- [ocp-schedules-28,7513d5e0563f5c3d,2952a76fe12ff163] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:43:17.102 INFO 26087 --- [ocp-schedules-28,7513d5e0563f5c3d,2952a76fe12ff163] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=7 2025-04-08 21:43:17.603 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:17.603 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:17.656 INFO 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:43:17.656 INFO 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:43:17.658 INFO 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:43:17.659 WARN 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:43:17.659 INFO 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:43:17.660 WARN 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:43:17.663 INFO 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:43:17.664 WARN 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:43:17.665 ERROR 26087 --- [ocp-schedules-25,c0430375e8899e74,e7dbc031d0fb5c2e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:43:18.244 INFO 26087 --- [ocp-schedules-19,969f729f24b0bab8,c8abba8f0e894e00] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:43:18.257 INFO 26087 --- [ocp-schedules-19,969f729f24b0bab8,c8abba8f0e894e00] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:43:18.257 INFO 26087 --- [ocp-schedules-19,969f729f24b0bab8,c8abba8f0e894e00] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:43:18.271 INFO 26087 --- [prometheus-minute-schedule-1,c0e838bc1f8ec791,2d1ed8aefdfc061a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:18.641 WARN 26087 --- [ocp-async-5,1f7b93f4610b54ec,7e4bff3a378b1940] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:18.641 INFO 26087 --- [ocp-async-2,6449d5b8331743ca,5143567d144b8361] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:19.635 INFO 26087 --- [ocp-schedules-21,5fd0386b3d398c73,6600d3d59962b412] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:43:19.636 INFO 26087 --- [ocp-schedules-21,5fd0386b3d398c73,6600d3d59962b412] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:43:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:43:20.603 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:20.604 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:20.641 INFO 26087 --- [ocp-async-6,854a0cbebe08c220,e08e069f493a8776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:20.641 WARN 26087 --- [ocp-async-1,dbffe0577f7d54b0,35873ef8b4234d5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:21.271 INFO 26087 --- [prometheus-minute-schedule-1,96f18d40e0c84862,ab910d8981ad02b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:22.641 WARN 26087 --- [ocp-async-6,9fe1ebd79f0734d6,83f7befe5a544312] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:22.641 INFO 26087 --- [ocp-async-9,88a6b45d8fe7b4da,26b42ad62ab296b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:22.880 INFO 26087 --- [ocp-schedules-21,b614223d5495b312,2ede1fc4a6169a65] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:43:23.604 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:23.605 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:24.271 INFO 26087 --- [prometheus-minute-schedule-1,5d3c875a4d036dbe,ea336f2eb2b99b7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:24.641 WARN 26087 --- [ocp-async-7,7323a9b05520f2d3,b25263cdd74b4a94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:24.641 INFO 26087 --- [ocp-async-5,84c121da1aa27f25,04e9d6ed33fe7656] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:26.442 INFO 26087 --- [ocp-schedules-27,bb0c22bb1a1ad66e,a0203037c236a909] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:43:26.442 INFO 26087 --- [ocp-schedules-7,75aa5bcef55fab13,cc41b6fe56621e3d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:43:26.466 INFO 26087 --- [ocp-schedules-29,5c80af7f0e3e9e82,05bb0eef3291d774] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 21:43:26.491 INFO 26087 --- [ocp-async-2,04d6c5240a70216c,cb185334909e191f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:43:26.605 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:26.605 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:43:26.622385+08:00, lastProcessTime=2025-04-08T21:43:16.623518+08:00 2025-04-08 21:43:26.641 WARN 26087 --- [ocp-async-4,fd9186f8b5b8d884,d27760d02a687047] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:26.641 INFO 26087 --- [ocp-async-8,1f1eaf3d0fca4a25,8f400061c377d168] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:26.736 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:43:26.622385+08:00 2025-04-08 21:43:26.765 INFO 26087 --- [ocp-schedules-27,dccb020122c463bc,a0aaddbebc18663e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:43:27.271 INFO 26087 --- [prometheus-minute-schedule-1,210d7edce03f1099,e34ca61958303a94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:28.641 WARN 26087 --- [ocp-async-3,3244af15b46e8a9f,9433ee8b95cac184] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:28.641 INFO 26087 --- [ocp-async-8,8c852bd06a7bd7e7,bed8ea613b38cace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:29.606 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:29.606 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:29.642 INFO 26087 --- [ocp-schedules-26,72aba0eb5e0e64de,15b593e39a12a8de] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:43:29.644 INFO 26087 --- [ocp-schedules-26,72aba0eb5e0e64de,15b593e39a12a8de] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:43:30.271 INFO 26087 --- [prometheus-minute-schedule-1,638bfa560678ee0b,18c4f35a4282b394] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:30.641 WARN 26087 --- [ocp-async-8,e6c9af3cffe58b66,b620d9a16ef56df5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:30.642 INFO 26087 --- [ocp-async-5,c6bc3b491aa9cdd6,6ce02cf1f27de71f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:32.606 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:32.607 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:32.642 WARN 26087 --- [ocp-async-5,32f3a243cafab037,c804ee5fbfce8820] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:32.642 INFO 26087 --- [ocp-async-4,aca450a044575672,496d522e9637eda4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:32.906 INFO 26087 --- [ocp-schedules-5,3f9ad5dd9ea14a02,12263feffb90741d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:43:33.271 INFO 26087 --- [prometheus-minute-schedule-1,6e9ec99c43798f2b,0886792b94bcd8f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:34.642 WARN 26087 --- [ocp-async-3,cd0599494f6ebbbd,51cb4b32b50bec45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:34.642 INFO 26087 --- [ocp-async-9,5856b5fbaaf12690,4f92086d7aef3417] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:34.895 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:43:34.895 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:43:34.896 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:43:34.896 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:43:34.897 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:43:34.901 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:43:34.901 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 21:43:35.607 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:35.607 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:36.271 INFO 26087 --- [prometheus-minute-schedule-1,89916612830faf59,c3cdda70e4c36c70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:36.491 INFO 26087 --- [ocp-async-10,6c9a1bf015188d17,061bd9e970e27ec9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:43:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:43:36.623569+08:00, lastProcessTime=2025-04-08T21:43:26.622385+08:00 2025-04-08 21:43:36.642 INFO 26087 --- [ocp-async-8,72d2791dc8c2e6ce,2eccecd51e364096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:36.642 WARN 26087 --- [ocp-async-2,56a969da18381e90,e825d2e79f376774] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:36.727 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:43:36.623569+08:00 2025-04-08 21:43:36.767 INFO 26087 --- [ocp-schedules-22,f4073cf6aad5ebbc,b6e4c4a99587b6c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:43:38.258 INFO 26087 --- [ocp-schedules-26,fdcc783bc5c4e26a,90b7a9ba6bbeb73b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:43:38.330 INFO 26087 --- [ocp-schedules-26,fdcc783bc5c4e26a,90b7a9ba6bbeb73b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:43:38.330 INFO 26087 --- [ocp-schedules-26,fdcc783bc5c4e26a,90b7a9ba6bbeb73b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:43:38.608 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:38.608 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:38.642 WARN 26087 --- [ocp-async-6,12923eb15d3a71da,039103e72a102e97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:38.642 INFO 26087 --- [ocp-async-5,afacba1a52896a6c,6d8a70f73e666952] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:39.271 INFO 26087 --- [prometheus-minute-schedule-1,6a082802f6da6504,17214e51fea0ce8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:39.649 INFO 26087 --- [ocp-schedules-20,0572005974a5c74c,e9f708bfe769dbe8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:43:39.651 INFO 26087 --- [ocp-schedules-20,0572005974a5c74c,e9f708bfe769dbe8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:43:40.002 INFO 26087 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:43:40.002 INFO 26087 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:43:40.642 WARN 26087 --- [ocp-async-3,18ff640c49353a5a,4a519ad99c7df19d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:40.642 INFO 26087 --- [ocp-async-10,4446a4421c4cc2f1,96648537c7256cd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:41.608 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:41.609 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:42.271 INFO 26087 --- [prometheus-minute-schedule-1,44af6a1cea0d3304,36ce93dd6931f047] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:42.642 WARN 26087 --- [ocp-async-2,97ea6f42cf32932b,c253ffdb2920a47d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:42.642 INFO 26087 --- [ocp-async-10,469d5935ee34a970,2656161f0c381ed8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:42.926 INFO 26087 --- [ocp-schedules-17,5b6eda74458a65ee,ff49e91ffbddcf3b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:43:44.609 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:44.609 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:44.643 INFO 26087 --- [ocp-async-8,baaf899489f47a12,0bcf4d427ac46872] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:44.643 WARN 26087 --- [ocp-async-4,3db1280c60dd4bfc,d71e6d75b0acd3f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:45.271 INFO 26087 --- [prometheus-minute-schedule-1,eea8e8405564d696,f131908d6df873b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:46.437 INFO 26087 --- [ocp-schedules-2,06342ebbfd088981,7b2017bb30261dab] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:43:46.437 INFO 26087 --- [ocp-schedules-24,4d1039d3795083d3,a237b2469d70ba7a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:43:46.438 INFO 26087 --- [ocp-schedules-24,4d1039d3795083d3,a237b2469d70ba7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:43:46.440 INFO 26087 --- [ocp-schedules-6,c1775fcfa7f14b2f,6b9e868194b73fe9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:43:46.440 INFO 26087 --- [ocp-schedules-6,c1775fcfa7f14b2f,6b9e868194b73fe9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:43:46.440 INFO 26087 --- [ocp-schedules-8,a74fcfad3863fb2b,e0e12bd35f3f9378] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:43:46.440 INFO 26087 --- [ocp-schedules-8,a74fcfad3863fb2b,e0e12bd35f3f9378] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:43:46.441 INFO 26087 --- [ocp-schedules-11,ab53236c8e0c9b9f,ce7fba697554f8b4] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:43:46.441 INFO 26087 --- [ocp-schedules-7,027bd63bcd0c2a4e,49a8578ef646a53c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:43:46.441 INFO 26087 --- [ocp-schedules-27,7bd677087e9a71df,9e2a57f9f89615eb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:43:46.442 INFO 26087 --- [ocp-schedules-27,7bd677087e9a71df,9e2a57f9f89615eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:46.442 INFO 26087 --- [ocp-schedules-7,027bd63bcd0c2a4e,49a8578ef646a53c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:43:46.492 INFO 26087 --- [ocp-async-6,4addd1ff80a198c9,c05404dc7abb63d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:43:46.568 INFO 26087 --- [ocp-schedules-15,e5c397d355085f96,1ccae33956291bde] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:43:46.568 INFO 26087 --- [ocp-schedules-15,bd30827beaf47dd8,4dcb396859162c21] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:43:46.569 INFO 26087 --- [ocp-schedules-15,bd30827beaf47dd8,4dcb396859162c21] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:43:46.574 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:43:46.574 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:43:46.586 INFO 26087 --- [ocp-schedules-14,bef30d6bef93b4a5,4e6cb964c14f5e6e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:43:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:43:46.623597+08:00, lastProcessTime=2025-04-08T21:43:36.623569+08:00 2025-04-08 21:43:46.643 WARN 26087 --- [ocp-async-7,94d2e089d5d06bdf,0a7e23b208e8146c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:46.643 INFO 26087 --- [ocp-async-9,6aa844566304d882,5adf413d0389c1e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:46.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:43:46.623597+08:00 2025-04-08 21:43:46.768 INFO 26087 --- [ocp-schedules-3,b0d845ddf5d18c81,6027e5e4af28cc99] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:43:47.610 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:47.610 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:48.271 INFO 26087 --- [prometheus-minute-schedule-1,5ddeca2e531708c5,5e0e003e0a393901] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:48.643 WARN 26087 --- [ocp-async-1,22fd6a6a689c1dea,f67c5944c5561910] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:48.643 INFO 26087 --- [ocp-async-2,10cf1aaada5b1360,71402aaf95e251e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:49.666 INFO 26087 --- [ocp-schedules-12,27b324b5efb0c01f,4ae8a9c29cf5aa16] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:43:49.667 INFO 26087 --- [ocp-schedules-12,27b324b5efb0c01f,4ae8a9c29cf5aa16] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:43:50.610 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:50.611 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:50.643 WARN 26087 --- [ocp-async-8,bf96d450e4c96569,f0ecd409667d51ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:50.643 INFO 26087 --- [ocp-async-4,7cf1cd2a72f950c4,c658058b92f33d4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:51.271 INFO 26087 --- [prometheus-minute-schedule-1,07ba5fec36c72cb7,ae173a56379fba8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:52.643 WARN 26087 --- [ocp-async-3,14f28466f43b79f0,c239ea61eb28ab7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:52.643 INFO 26087 --- [ocp-async-7,9c3ec64d6a0b8cb5,1f982c6cccd19561] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:52.950 INFO 26087 --- [ocp-schedules-22,19bae856b0f0b15b,526cbc4eee856452] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:43:53.611 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:53.611 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:54.271 INFO 26087 --- [prometheus-minute-schedule-1,86b11eb52cd52a1a,b44def6ae487487d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:54.643 WARN 26087 --- [ocp-async-1,07c6778eeb7203ac,60cd7629d958f986] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:54.643 INFO 26087 --- [ocp-async-9,1d9ca13dca59b2ae,d8093ce0848be8ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:56.442 INFO 26087 --- [ocp-schedules-27,7bd677087e9a71df,9e2a57f9f89615eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:43:56.442 INFO 26087 --- [ocp-schedules-7,027bd63bcd0c2a4e,49a8578ef646a53c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:43:56.492 INFO 26087 --- [ocp-async-10,976cfb48823adeef,12368f6d8e9f8969] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:43:56.611 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:56.612 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:43:56.623800+08:00, lastProcessTime=2025-04-08T21:43:46.623597+08:00 2025-04-08 21:43:56.643 WARN 26087 --- [ocp-async-4,33051d5bbaac8742,55f50b0f241a681e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:56.643 INFO 26087 --- [ocp-async-8,f9e2b76e7af2d645,4b7c54370f2cdd35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:56.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:43:56.623800+08:00 2025-04-08 21:43:56.769 INFO 26087 --- [ocp-schedules-2,ea3f822c3b9935e4,0cf7901636cb354e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:43:57.271 INFO 26087 --- [prometheus-minute-schedule-1,0f9fbb158c3eae7c,53f640fa8a1b7d77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119840, interval=60 2025-04-08 21:43:58.331 INFO 26087 --- [ocp-schedules-23,0384d88bafc677d0,cc34a216e11f449d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:43:58.335 INFO 26087 --- [ocp-schedules-23,0384d88bafc677d0,cc34a216e11f449d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:43:58.335 INFO 26087 --- [ocp-schedules-23,0384d88bafc677d0,cc34a216e11f449d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:43:58.644 WARN 26087 --- [ocp-async-8,6fda485fb6fae6ad,2cafb221774610ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:43:58.644 INFO 26087 --- [ocp-async-3,462043638a969c7b,bad6389e4101ad2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:43:59.612 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:43:59.613 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:43:59.697 INFO 26087 --- [ocp-schedules-12,224e88a2b75c1bc2,b76e048ee2e641e1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:43:59.698 INFO 26087 --- [ocp-schedules-12,224e88a2b75c1bc2,b76e048ee2e641e1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:44:00.271 INFO 26087 --- [prometheus-minute-schedule-1,a7a22b2fc5e45683,6ef806cc4be5c0f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:00.644 INFO 26087 --- [ocp-async-3,e77586ffc0ef137f,44294b3d0b97394f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:00.644 WARN 26087 --- [ocp-async-10,11958496fa9e1eef,3bbae5d2bbe8ae3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:02.613 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:02.613 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:02.644 WARN 26087 --- [ocp-async-2,6e9bf0d3cd370188,a7fb0eefcd76337b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:02.644 INFO 26087 --- [ocp-async-4,9e174e21a2125c58,b3f6c97225f5bf83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:02.971 INFO 26087 --- [ocp-schedules-13,14a9b27d74fc632b,2a2898adecc0372c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:44:03.271 INFO 26087 --- [prometheus-minute-schedule-1,b9c1db4ef3496ee3,8f0a95c96bed1ce2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:04.644 WARN 26087 --- [ocp-async-8,6e4982455fb05d27,cd7a80355fd17386] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:04.644 INFO 26087 --- [ocp-async-7,3b76c683b3581185,c013463978b4e86a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:05.614 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:05.614 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:06.271 INFO 26087 --- [prometheus-minute-schedule-1,a123c876cf993e38,cafe1edd93c30594] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:06.272 INFO 26087 --- [prometheus-cache-monitor1,0a789df2cfdf0aae,441bd248cd02d01a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:44:06.272 INFO 26087 --- [prometheus-cache-monitor1,0a789df2cfdf0aae,441bd248cd02d01a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:44:06.452 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:38:06.337, end=2025-04-08T21:43:06.409} 2025-04-08 21:44:06.453 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:38:06.34, end=2025-04-08T21:43:06.451} 2025-04-08 21:44:06.456 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:44:06.456 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:44:06.456 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:44:06.456 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:44:06.492 INFO 26087 --- [ocp-async-5,222a4d1fd0cb98d5,44596b7df581c4e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:44:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:44:06.622235+08:00, lastProcessTime=2025-04-08T21:43:56.623800+08:00 2025-04-08 21:44:06.644 WARN 26087 --- [ocp-async-10,6988684aa7ae40f3,7c3d88ff485ff094] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:06.644 INFO 26087 --- [ocp-async-9,7dd0ab914d20a935,24a0da3777deba93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:06.712 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:44:06.622235+08:00 2025-04-08 21:44:06.771 INFO 26087 --- [ocp-schedules-11,5402efeee6ca506a,90029317c06e6100] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:44:06.866 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:40:06.806, end=2025-04-08T21:42:06.859} 2025-04-08 21:44:06.868 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:44:06.868 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:44:08.614 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:08.615 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:08.644 WARN 26087 --- [ocp-async-9,def9a5d4d6b631d3,ea5c9ed5f4351420] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:08.644 INFO 26087 --- [ocp-async-3,bd1c85e05a76d81a,6be13a97807d1622] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:09.271 INFO 26087 --- [prometheus-minute-schedule-1,5608699d7b40fb62,0150cfdd20693680] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:09.701 INFO 26087 --- [ocp-schedules-1,350dd6eabc1a9620,2aa156d4989e67be] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:44:09.702 INFO 26087 --- [ocp-schedules-1,350dd6eabc1a9620,2aa156d4989e67be] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:44:10.001 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:44:10.001 INFO 26087 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:44:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:44:10.644 WARN 26087 --- [ocp-async-4,28757b984382c981,fa5beedccb80229b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:10.644 INFO 26087 --- [ocp-async-3,924d67d60610a3cb,2ec2656bdb713d8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:11.615 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:11.615 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:12.271 INFO 26087 --- [prometheus-minute-schedule-1,917665f0f3fd00ec,c52fba264ee5057a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:12.555 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119780, nodeCount=78, affectRows=78 2025-04-08 21:44:12.555 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119780, nodeCount=148, affectRows=148 2025-04-08 21:44:12.555 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119780, nodeCount=119, affectRows=119 2025-04-08 21:44:12.555 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119780, nodeCount=35, affectRows=35 2025-04-08 21:44:12.555 INFO 26087 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119780, nodeCount=35, validCount=35 2025-04-08 21:44:12.556 INFO 26087 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119780, nodeCount=78, validCount=78 2025-04-08 21:44:12.556 INFO 26087 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119780, nodeCount=118, validCount=118 2025-04-08 21:44:12.556 INFO 26087 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119780, nodeCount=146, validCount=146 2025-04-08 21:44:12.645 WARN 26087 --- [ocp-async-5,bc753104b9124b51,77c7dc7c93e8b5cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:12.645 INFO 26087 --- [ocp-async-1,9e310a895a3e62b2,f3bd5bc51900e91a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:12.995 INFO 26087 --- [ocp-schedules-7,0ee423db165f32fb,50be4bcbaefef823] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:44:14.616 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:14.616 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:14.645 WARN 26087 --- [ocp-async-9,9756d0f0a6989613,49e1df1f8996d02b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:14.645 INFO 26087 --- [ocp-async-10,79b2ec77a3b8efe3,c7cb3e52c8dd61c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:15.271 INFO 26087 --- [prometheus-minute-schedule-1,923113e6e08c94fe,14dae85e726be094] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:16.433 INFO 26087 --- [ocp-schedules-7,d98547b488c3c90a,2a0a6900a4f02a55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:44:16.434 INFO 26087 --- [ocp-schedules-22,a4bdaaae70820e77,ab493d239fb30ff0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:44:16.437 INFO 26087 --- [ocp-schedules-7,d98547b488c3c90a,2a0a6900a4f02a55] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:44:16.437 INFO 26087 --- [ocp-schedules-9,d1d1d1216fcf6946,fb6a1eb82922b719] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:44:16.437 INFO 26087 --- [ocp-schedules-30,f7b6cc660b311153,8fb88796b57cf22c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:44:16.437 INFO 26087 --- [ocp-schedules-30,f7b6cc660b311153,8fb88796b57cf22c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:44:16.439 INFO 26087 --- [ocp-schedules-14,8aa8cf1eb56e7b34,ec1c03aea9f30e5d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:44:16.440 INFO 26087 --- [ocp-schedules-14,8aa8cf1eb56e7b34,ec1c03aea9f30e5d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:44:16.440 INFO 26087 --- [ocp-schedules-13,2ffce2368cc4a918,32410ef7575068fc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:44:16.440 INFO 26087 --- [ocp-schedules-13,2ffce2368cc4a918,32410ef7575068fc] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:44:16.441 INFO 26087 --- [ocp-schedules-24,8473cb01d81bdc52,a50c8e9876d885eb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:44:16.441 INFO 26087 --- [ocp-schedules-16,574eb0eea6efe15c,7db1f995af0e8000] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:44:16.441 INFO 26087 --- [ocp-schedules-24,8473cb01d81bdc52,a50c8e9876d885eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:16.442 INFO 26087 --- [ocp-schedules-16,574eb0eea6efe15c,7db1f995af0e8000] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:44:16.442 INFO 26087 --- [ocp-schedules-22,a4bdaaae70820e77,ab493d239fb30ff0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:44:16.446 INFO 26087 --- [ocp-schedules-19,a22e51567cb42d0c,886bb52d5742465a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:44:16.480 INFO 26087 --- [ocp-schedules-5,9ec353bb7e3dfc1d,0c39376c41fa50bf] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-04-08 21:44:16.480 INFO 26087 --- [ocp-schedules-5,9ec353bb7e3dfc1d,0c39376c41fa50bf] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-04-08 21:44:16.492 INFO 26087 --- [ocp-async-4,a9c1567442b8b2c5,1dc63c804864fd30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:44:16.509 INFO 26087 --- [ocp-schedules-4,f1be85a9855a7822,41e90c606f3d2009] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:44:16.509 INFO 26087 --- [ocp-schedules-4,f1be85a9855a7822,41e90c606f3d2009] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:16.509 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:44:16.541 INFO 26087 --- [ocp-schedules-23,52cadda21f04119f,e864f1ed7e135707] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:44:16.575 INFO 26087 --- [ocp-schedules-15,bd30827beaf47dd8,4dcb396859162c21] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:44:16.575 INFO 26087 --- [ocp-schedules-15,74ffbaeeebc58b2a,eb314c1fa3ceba64] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:44:16.576 INFO 26087 --- [ocp-schedules-15,74ffbaeeebc58b2a,eb314c1fa3ceba64] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:16.581 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:44:16.581 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:44:16.587 INFO 26087 --- [ocp-schedules-27,05b9f18bee8e32f7,bfee48dbd8b9c3a4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:44:16.588 INFO 26087 --- [ocp-schedules-14,78e43fed73bc8034,5c2992b939da5316] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:44:16.588 INFO 26087 --- [ocp-schedules-14,78e43fed73bc8034,5c2992b939da5316] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:44:16.589 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:44:16.589 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:44:16.594 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:44:16.594 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:44:16.594 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:44:16.596 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:44:16.596 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:44:16.596 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:44:16.598 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:44:16.598 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:44:16.598 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:44:16.599 INFO 26087 --- [ocp-schedules-9,96e6c5868c760bd8,a4e81ebf7c2184b6] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:44:16.599 INFO 26087 --- [ocp-schedules-9,96e6c5868c760bd8,a4e81ebf7c2184b6] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:44:16.600 INFO 26087 --- [ocp-schedules-13,13829c7cd6fbf78e,dca5423ec44fef82] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:44:16.624 INFO 26087 --- [ocp-schedules-22,059e02fb5f5793df,cbb8ffc7f95dc343] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:44:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:44:16.623978+08:00, lastProcessTime=2025-04-08T21:44:06.622235+08:00 2025-04-08 21:44:16.625 INFO 26087 --- [ocp-schedules-22,059e02fb5f5793df,cbb8ffc7f95dc343] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:44:16.625 INFO 26087 --- [ocp-schedules-22,059e02fb5f5793df,cbb8ffc7f95dc343] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:44:16.633 INFO 26087 --- [ocp-schedules-19,e8af21505b4338c8,f00855291b401a22] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:44:16.633 INFO 26087 --- [ocp-schedules-7,fdf64c78c119ab00,bbd174fe77dbe330] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:44:16.633+08:00, clusters=[1000003] 2025-04-08 21:44:16.633 INFO 26087 --- [ocp-schedules-7,fdf64c78c119ab00,bbd174fe77dbe330] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:44:16.644 INFO 26087 --- [ocp-schedules-17,1848acec2be4d9a5,a620aa852bd51be0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:44:16.644+08:00, clusters=[1000003] 2025-04-08 21:44:16.645 WARN 26087 --- [ocp-async-3,253f0268dbd6fad9,1b55b601f2926eff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:16.645 INFO 26087 --- [ocp-async-2,79a588b572d68498,32c3cd2bcda93e73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:16.646 INFO 26087 --- [ocp-schedules-17,1848acec2be4d9a5,a620aa852bd51be0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:44:16.650 INFO 26087 --- [ocp-schedules-2,bc35948923a43f64,49b4d35c72f96b50] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:44:16.650 INFO 26087 --- [ocp-schedules-2,bc35948923a43f64,49b4d35c72f96b50] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:44:16.654 INFO 26087 --- [ocp-schedules-2,bc35948923a43f64,49b4d35c72f96b50] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:44:16.654 INFO 26087 --- [ocp-schedules-2,bc35948923a43f64,49b4d35c72f96b50] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:44:16.713 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:44:16.623978+08:00 2025-04-08 21:44:16.772 INFO 26087 --- [ocp-schedules-6,c856ce27af782447,5c094d7def187b80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:44:16.844 INFO 26087 --- [ocp-schedules-11,8030634734e4cf79,97a69221f6996e49] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:44:16.844 INFO 26087 --- [ocp-schedules-11,8030634734e4cf79,97a69221f6996e49] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:44:16.844 INFO 26087 --- [ocp-schedules-11,8030634734e4cf79,97a69221f6996e49] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:44:16.844 INFO 26087 --- [ocp-schedules-11,8030634734e4cf79,97a69221f6996e49] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:44:16.844 INFO 26087 --- [ocp-schedules-11,8030634734e4cf79,97a69221f6996e49] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:44:16.844 INFO 26087 --- [ocp-schedules-11,8030634734e4cf79,97a69221f6996e49] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:44:16.845 INFO 26087 --- [ocp-schedules-21,ce0e3fe9dfbe9c16,248e19eed551932a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:44:16.846 INFO 26087 --- [ocp-schedules-21,ce0e3fe9dfbe9c16,248e19eed551932a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:16.857 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:44:16.858 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:44:16.949 INFO 26087 --- [ocp-schedules-25,5710df6197493095,983e38673a274ac7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:44:16.950 INFO 26087 --- [ocp-schedules-25,5710df6197493095,983e38673a274ac7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:16.965 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:44:16.966 ERROR 26087 --- [ocp-schedules-25,5710df6197493095,983e38673a274ac7] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:44:16.966 INFO 26087 --- [ocp-schedules-25,5710df6197493095,983e38673a274ac7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:44:17.053 INFO 26087 --- [ocp-schedules-28,e0b21a192df81a75,9be2c9edddc8504f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:44:17.616 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:17.617 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:17.668 INFO 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:44:17.668 INFO 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:44:17.671 INFO 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:44:17.672 WARN 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:44:17.672 INFO 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:44:17.673 WARN 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:44:17.676 INFO 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:44:17.677 WARN 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:44:17.678 ERROR 26087 --- [ocp-schedules-13,87e9eb653f1ae42a,9d74bf4d8fa09709] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:44:18.271 INFO 26087 --- [prometheus-minute-schedule-1,6688413703fae488,7b008fbc5059df40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:18.336 INFO 26087 --- [ocp-schedules-26,74c550b8b5b9b7f5,fd004e68755f5d52] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:44:18.343 INFO 26087 --- [ocp-schedules-26,74c550b8b5b9b7f5,fd004e68755f5d52] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:44:18.343 INFO 26087 --- [ocp-schedules-26,74c550b8b5b9b7f5,fd004e68755f5d52] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:44:18.645 WARN 26087 --- [ocp-async-1,22b2362f7e01b137,c2d297df0341adf9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:18.645 INFO 26087 --- [ocp-async-8,ced9c41c8fadd854,cfb98e7849e73b32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:19.762 INFO 26087 --- [ocp-schedules-11,dac9b66592733f0f,351f71f85433e1d1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:44:19.764 INFO 26087 --- [ocp-schedules-11,dac9b66592733f0f,351f71f85433e1d1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:44:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:44:20.617 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:20.617 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:20.645 WARN 26087 --- [ocp-async-10,1a27b37525a992be,ad709c4429a4f655] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:20.645 INFO 26087 --- [ocp-async-4,fc1a2c53c6e19f4c,7f16c96c322938ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:21.271 INFO 26087 --- [prometheus-minute-schedule-1,0592ca7539d28058,54cc733bd489ef5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:22.645 WARN 26087 --- [ocp-async-5,94147dc6c863737e,075e1201b69bb512] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:22.645 INFO 26087 --- [ocp-async-3,debf22ca149e2352,af889803d2d5ab4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:23.014 INFO 26087 --- [ocp-schedules-3,2e155f5fdbe977d1,70d93c2b3de29beb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:44:23.617 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:23.618 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:24.271 INFO 26087 --- [prometheus-minute-schedule-1,c9fd04a5a59a22e5,e959064daeba90a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:24.645 WARN 26087 --- [ocp-async-9,6b4a0bacd11f1d0e,c48e144e51884d8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:24.645 INFO 26087 --- [ocp-async-1,8178e111ce977c53,60670e127ea61281] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:26.442 INFO 26087 --- [ocp-schedules-24,8473cb01d81bdc52,a50c8e9876d885eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:44:26.442 INFO 26087 --- [ocp-schedules-16,574eb0eea6efe15c,7db1f995af0e8000] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:44:26.492 INFO 26087 --- [ocp-async-8,55a2df3ff502a412,e6e8d3fc50be6210] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:44:26.510 INFO 26087 --- [ocp-schedules-4,f1be85a9855a7822,41e90c606f3d2009] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:44:26.618 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:26.619 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:44:26.623973+08:00, lastProcessTime=2025-04-08T21:44:16.623978+08:00 2025-04-08 21:44:26.646 INFO 26087 --- [ocp-async-4,01b1ca5fd8c91a85,36c3754bd236a839] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:26.646 WARN 26087 --- [ocp-async-5,4aef54a539e74048,370b07bc5ff43dde] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:26.733 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:44:26.623973+08:00 2025-04-08 21:44:26.774 INFO 26087 --- [ocp-schedules-27,858d776972a14722,aa90bccd04b29d79] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:44:27.271 INFO 26087 --- [prometheus-minute-schedule-1,dafbc37380db27bd,d606afd0adc8a05d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:28.646 WARN 26087 --- [ocp-async-9,dccdfb4890cf3549,c07c70574bbe7078] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:28.646 INFO 26087 --- [ocp-async-2,eea8ac7cdce1912c,c3ef5c0e206b2d66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:29.619 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:29.619 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:29.809 INFO 26087 --- [ocp-schedules-5,375ce8d27c3bfea7,c2d7d275523fffc6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:44:29.810 INFO 26087 --- [ocp-schedules-5,375ce8d27c3bfea7,c2d7d275523fffc6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:44:30.271 INFO 26087 --- [prometheus-minute-schedule-1,4a3d00fe648756b1,fdcf6bf43cb49fff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:30.646 WARN 26087 --- [ocp-async-2,53f9167cd19ff5c6,cca6ebbcff59ad76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:30.646 INFO 26087 --- [ocp-async-1,4e02b36f8e7ec0ee,4ae6d1a058b6facb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:32.619 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:32.620 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:32.646 WARN 26087 --- [ocp-async-6,b4c9827f0765a193,148e44e5b4b95c79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:32.646 INFO 26087 --- [ocp-async-1,46e932623cdd5b70,2ec176653a135c56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:33.057 INFO 26087 --- [ocp-schedules-4,500e8c4cffcd2d54,39c2b025735174a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:44:33.271 INFO 26087 --- [prometheus-minute-schedule-1,bbf3638486dc3b58,ddbf570645095e31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:34.646 INFO 26087 --- [ocp-async-9,8179f39950494c43,3a4fe1ee4b7475ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:34.646 WARN 26087 --- [ocp-async-1,3858e065063ff8e8,826244f15fb30e04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:35.620 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:35.621 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:36.271 INFO 26087 --- [prometheus-minute-schedule-1,ea3c2c9be51b32ff,4bcba770f31f0e87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:36.492 INFO 26087 --- [ocp-async-4,839b1ea625f2789d,9a6e1122395dc962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:44:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:44:36.622971+08:00, lastProcessTime=2025-04-08T21:44:26.623973+08:00 2025-04-08 21:44:36.646 WARN 26087 --- [ocp-async-2,1072a84e445e0e0a,180c8e173e9df0d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:36.646 INFO 26087 --- [ocp-async-8,dff3a23f8238c54e,ce86ada8a456fe6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:44:36.622971+08:00 2025-04-08 21:44:36.775 INFO 26087 --- [ocp-schedules-8,79976ef325faf0bf,77cc7b5ed65d3348] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:44:38.344 INFO 26087 --- [ocp-schedules-8,2051b47fa09f6e03,8ea598e5efb27e28] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:44:38.384 INFO 26087 --- [ocp-schedules-8,2051b47fa09f6e03,8ea598e5efb27e28] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:44:38.384 INFO 26087 --- [ocp-schedules-8,2051b47fa09f6e03,8ea598e5efb27e28] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:44:38.621 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:38.621 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:38.646 WARN 26087 --- [ocp-async-7,ff6bdccb5659766e,87d37a335518246a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:38.647 INFO 26087 --- [ocp-async-6,731e385c3e53f429,3bd0442901759c5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:39.271 INFO 26087 --- [prometheus-minute-schedule-1,9525e0879b115411,84be1ca47b40fdba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:39.862 INFO 26087 --- [ocp-schedules-27,2b5fba45895e16c6,b945593ad34a9b5d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:44:39.863 INFO 26087 --- [ocp-schedules-27,2b5fba45895e16c6,b945593ad34a9b5d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:44:40.002 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:44:40.002 INFO 26087 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:44:40.647 INFO 26087 --- [ocp-async-9,53d9b5c31cd8ba37,6ec13174aac33738] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:40.647 WARN 26087 --- [ocp-async-4,71915f9a6703d6fd,c9b6084e4f9320cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:41.621 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:41.622 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:42.271 INFO 26087 --- [prometheus-minute-schedule-1,3498f3c37329d09d,5b7f131906901782] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:42.647 WARN 26087 --- [ocp-async-10,920b5bafb49f037b,9d31d3c04afab62d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:42.647 INFO 26087 --- [ocp-async-9,10309721423218f6,7375863b7e5dea2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:43.077 INFO 26087 --- [ocp-schedules-26,20a26e65b2318f7e,ead34447ca0aeb50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:44:44.622 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:44.623 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:44.647 WARN 26087 --- [ocp-async-8,6c2ca2a6f38c2556,ecde634f89912bfb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:44.647 INFO 26087 --- [ocp-async-5,4744f1fac966bff0,5b0be378d3066002] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:45.271 INFO 26087 --- [prometheus-minute-schedule-1,8a4050d18d3d955d,a1e1fcf316ed20b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:46.437 INFO 26087 --- [ocp-schedules-16,09b56a849f6538a3,f68352aed5bc031e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:44:46.437 INFO 26087 --- [ocp-schedules-3,917b874c9329ddf7,e79589a86a3e350c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:44:46.437 INFO 26087 --- [ocp-schedules-3,917b874c9329ddf7,e79589a86a3e350c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:44:46.439 INFO 26087 --- [ocp-schedules-13,c39e68d48c4c6611,0949df5c4b79fff7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:44:46.440 INFO 26087 --- [ocp-schedules-13,c39e68d48c4c6611,0949df5c4b79fff7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:44:46.440 INFO 26087 --- [ocp-schedules-6,722951f237683332,7df6098b9dee3d14] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:44:46.440 INFO 26087 --- [ocp-schedules-6,722951f237683332,7df6098b9dee3d14] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:44:46.440 INFO 26087 --- [ocp-schedules-3,5b97969d52e2fad6,47843fccaa666de6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:44:46.441 INFO 26087 --- [ocp-schedules-7,fa41362e4693d40c,652991d53cf1b73c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:44:46.441 INFO 26087 --- [ocp-schedules-23,3c94b51865f0a188,0d00e14b1b3957b8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:44:46.441 INFO 26087 --- [ocp-schedules-7,fa41362e4693d40c,652991d53cf1b73c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:46.441 INFO 26087 --- [ocp-schedules-23,3c94b51865f0a188,0d00e14b1b3957b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:44:46.493 INFO 26087 --- [ocp-async-7,45186636e829139d,372488c769a4eada] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:44:46.581 INFO 26087 --- [ocp-schedules-15,74ffbaeeebc58b2a,eb314c1fa3ceba64] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:44:46.581 INFO 26087 --- [ocp-schedules-15,306e14c2c2821704,9ea00ea801afea9e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:44:46.582 INFO 26087 --- [ocp-schedules-15,306e14c2c2821704,9ea00ea801afea9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:44:46.587 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:44:46.587 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:44:46.588 INFO 26087 --- [ocp-schedules-10,2f2e90b8c2e70ee3,29309117d17f943a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:44:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:44:46.622532+08:00, lastProcessTime=2025-04-08T21:44:36.622971+08:00 2025-04-08 21:44:46.647 WARN 26087 --- [ocp-async-6,171a57d0d53b24e6,af295f37e6a780d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:46.647 INFO 26087 --- [ocp-async-3,2d3e9695c12ee7d2,8d2d1df94b1ccdd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:46.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:44:46.622532+08:00 2025-04-08 21:44:46.776 INFO 26087 --- [ocp-schedules-11,06c29062888aa91c,9b0af1d04cf0ad0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:44:47.623 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:47.623 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:48.271 INFO 26087 --- [prometheus-minute-schedule-1,fad1ee49e93b3348,bb883d8b0f7927dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:48.647 WARN 26087 --- [ocp-async-2,7b7bf694e3ce2b34,df26f4cec0dbee42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:48.647 INFO 26087 --- [ocp-async-10,c251d074afc4b514,0de5ec6090444521] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:49.895 INFO 26087 --- [ocp-schedules-11,2f5d6e04bbe42d64,b76ec1b041ff864e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:44:49.896 INFO 26087 --- [ocp-schedules-11,2f5d6e04bbe42d64,b76ec1b041ff864e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:44:50.623 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:50.624 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:50.647 WARN 26087 --- [ocp-async-10,e4dac79b0460ac4f,f7fb28ef5c808495] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:50.647 INFO 26087 --- [ocp-async-5,7a89aeba1f581b13,1f531954c6e57881] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:51.271 INFO 26087 --- [prometheus-minute-schedule-1,5ef5d196332a17ab,2a03c2d1d88c7370] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:52.647 WARN 26087 --- [ocp-async-6,7d68f3e33cba9678,77bea538f1af35bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:52.647 INFO 26087 --- [ocp-async-1,44035d886050950e,80ed5a73df061f35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:53.107 INFO 26087 --- [ocp-schedules-24,d25d2d6726a7c088,e360cf148c31547b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:44:53.624 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:53.625 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:54.271 INFO 26087 --- [prometheus-minute-schedule-1,4e6262a50f53db41,41cd4198552cc597] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:54.648 WARN 26087 --- [ocp-async-9,54341bf5fc6efe3d,63706bef9a1203c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:54.648 INFO 26087 --- [ocp-async-3,160a747c57a27ebf,881e8c1d55c3434b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:56.442 INFO 26087 --- [ocp-schedules-23,3c94b51865f0a188,0d00e14b1b3957b8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:44:56.442 INFO 26087 --- [ocp-schedules-7,fa41362e4693d40c,652991d53cf1b73c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:44:56.493 INFO 26087 --- [ocp-async-2,20dd9d922f8aeda6,1991db70a70fda6c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:44:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:44:56.623835+08:00, lastProcessTime=2025-04-08T21:44:46.622532+08:00 2025-04-08 21:44:56.625 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:56.625 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:56.648 WARN 26087 --- [ocp-async-5,74c7358772dbb9c9,08a3f14776239a4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:56.648 INFO 26087 --- [ocp-async-10,32dd3cd991ca7506,a3041d3a450805b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:56.721 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:44:56.623835+08:00 2025-04-08 21:44:56.778 INFO 26087 --- [ocp-schedules-18,a6e94cdf27703097,e3f17c1c9fbd604b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:44:57.271 INFO 26087 --- [prometheus-minute-schedule-1,590a16cccf555b0e,26a6ac55e792cea3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119900, interval=60 2025-04-08 21:44:58.385 INFO 26087 --- [ocp-schedules-18,d8666b0074058b54,8c2137768ba1f97e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:44:58.389 INFO 26087 --- [ocp-schedules-18,d8666b0074058b54,8c2137768ba1f97e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:44:58.389 INFO 26087 --- [ocp-schedules-18,d8666b0074058b54,8c2137768ba1f97e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:44:58.648 INFO 26087 --- [ocp-async-6,cf94fbc8b2640c66,cc23b2edd99fade6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:44:58.648 WARN 26087 --- [ocp-async-1,f4a2839f28d8f76b,a2908e08124d63c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:44:59.625 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:44:59.626 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:44:59.922 INFO 26087 --- [ocp-schedules-8,9c4b74d07a5a5daf,be3bff05339991ba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:44:59.923 INFO 26087 --- [ocp-schedules-8,9c4b74d07a5a5daf,be3bff05339991ba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:45:00.045 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:40:00.0, nextRun=2025-04-08 21:45:00.0, currentDate=Tue Apr 08 21:45:00 CST 2025 2025-04-08 21:45:00.045 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:40:00.0, nextRunTime=2025-04-08 21:45:00.0} 2025-04-08 21:45:00.047 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-04-08 21:45:00.048 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='192.168.1.61', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-04-08T21:45:00.047+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8ec1a44aeefd3a62', operation=EXECUTE} 2025-04-08 21:45:00.098 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-04-08 21:40:00.0, nextRun=2025-04-08 21:45:00.0, currentDate=Tue Apr 08 21:45:00 CST 2025 2025-04-08 21:45:00.098 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-04-08 21:40:00.0, nextRunTime=2025-04-08 21:45:00.0} 2025-04-08 21:45:00.102 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[] 2025-04-08 21:45:00.105 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.p.s.internal.SqlParseTaskSchedule : No tenant found, skip current task 2025-04-08 21:45:00.105 INFO 26087 --- [task-coordinator2,8ec1a44aeefd3a62,8445e7592025132a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2025-04-08 21:45:00.271 INFO 26087 --- [prometheus-minute-schedule-1,945b3133eae05522,4bb65c766abb8cfb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:00.648 WARN 26087 --- [ocp-async-9,3e18e63816614e25,ad8c1700e3ae0b55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:00.648 INFO 26087 --- [ocp-async-2,210262c2393fa618,5ee683bc67c41950] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:00.689 INFO 26087 --- [subtask-coordinator2,8ec1a44aeefd3a62,4a5fbe83a80c083a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000167, operation:EXECUTE, state: RUNNING 2025-04-08 21:45:00.706 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-04-08 21:45:00.707 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=1000167, context=Context{parallelIdx=-1, stringMap={task_instance_id=1000020, task_operation=execute, latest_execution_start_time=2025-04-08T21:45:00.685+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=1000167}, listMap={}}, executor=192.168.1.61 2025-04-08 21:45:00.710 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.711 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.o.DiskUsageTimeRemainingOperator : no metrics data, skip predicting for ocpdb 2025-04-08 21:45:00.711 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.711 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.712 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.712 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.712 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.713 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.713 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:ocpdb 2025-04-08 21:45:00.723 INFO 26087 --- [subtask-executor16,8ec1a44aeefd3a62,93a357480da96136] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 1000167, operation:EXECUTE, state: SUCCESSFUL 2025-04-08 21:45:01.387 INFO 26087 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:1000020, status:SUCCESS 2025-04-08 21:45:01.387 INFO 26087 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:1000020, status:SUCCESS 2025-04-08 21:45:02.626 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:02.627 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:02.648 WARN 26087 --- [ocp-async-2,82737fef298ecb2c,4a37428611fa6975] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:02.648 INFO 26087 --- [ocp-async-3,7eaf16caf67ea3fd,a609c5ba8e046717] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:03.128 INFO 26087 --- [ocp-schedules-8,db4bcc62d44ac1a6,a25a9256103bac1b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:45:03.271 INFO 26087 --- [prometheus-minute-schedule-1,78308d0934959db4,9aca357139b61d6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:04.648 INFO 26087 --- [ocp-async-3,94aa4601f33918ff,f952ce3e21441e5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:04.648 WARN 26087 --- [ocp-async-1,4d562bc553a4e71d,b1c3f22ab5b2b22a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:05.627 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:05.627 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:06.271 INFO 26087 --- [prometheus-minute-schedule-1,f988b204055224f4,7011a691d483f1d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:06.272 INFO 26087 --- [prometheus-cache-monitor1,3250297668bf1cc0,3cc1878065919b86] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:45:06.272 INFO 26087 --- [prometheus-cache-monitor1,3250297668bf1cc0,3cc1878065919b86] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:45:06.493 INFO 26087 --- [ocp-async-10,6f91961be9c42c9d,4e2592bfda21e678] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:45:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:45:06.623422+08:00, lastProcessTime=2025-04-08T21:44:56.623835+08:00 2025-04-08 21:45:06.649 WARN 26087 --- [ocp-async-8,1033d3593917f4e8,59de933bc3fa44a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:06.649 INFO 26087 --- [ocp-async-6,421a39e6d21d4421,280617fe51f90670] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:06.727 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:45:06.623422+08:00 2025-04-08 21:45:06.779 INFO 26087 --- [ocp-schedules-28,d7a6b7d43be79b6e,0c8274f15fc96211] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:45:08.627 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:08.628 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:08.649 WARN 26087 --- [ocp-async-7,ba2bbf1798302fd2,2daa2cf9448f764b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:08.649 INFO 26087 --- [ocp-async-5,f97904f8e9c46052,691df2835da17676] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:09.271 INFO 26087 --- [prometheus-minute-schedule-1,05660a8b2b0ab93b,33d36a97e8c40b3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:09.941 INFO 26087 --- [ocp-schedules-18,7c67421f18a05462,b8c69d5bf619031d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:45:09.942 INFO 26087 --- [ocp-schedules-18,7c67421f18a05462,b8c69d5bf619031d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:45:10.001 INFO 26087 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:45:10.001 INFO 26087 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:45:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:45:10.649 INFO 26087 --- [ocp-async-4,a04c628cddad3cab,07b11bf3bf6d5627] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:10.649 WARN 26087 --- [ocp-async-10,0050c64f5c506318,47350881599e244d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:11.628 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:11.629 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:12.271 INFO 26087 --- [prometheus-minute-schedule-1,55b705b47fabaa83,779a8c6bdf70a371] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:12.494 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119840, nodeCount=124, affectRows=124 2025-04-08 21:45:12.495 INFO 26087 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119840, nodeCount=123, validCount=123 2025-04-08 21:45:12.540 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119840, nodeCount=123, affectRows=123 2025-04-08 21:45:12.540 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119840, nodeCount=133, affectRows=133 2025-04-08 21:45:12.541 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119840, nodeCount=122, validCount=122 2025-04-08 21:45:12.541 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119840, nodeCount=132, validCount=132 2025-04-08 21:45:12.649 WARN 26087 --- [ocp-async-9,ff5fe0e65afd8158,c493f0037e561966] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:12.649 INFO 26087 --- [ocp-async-4,ae0c2321ecebfacd,7ed45fbec77a9c19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:13.152 INFO 26087 --- [ocp-schedules-4,b54f02e85174bef5,6428bf7ab334d70d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:45:14.629 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:14.629 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:14.649 WARN 26087 --- [ocp-async-7,c0f34f59fcd7a5e5,71bbdc2708e9460f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:14.649 INFO 26087 --- [ocp-async-4,eb663c6f79013c5e,0982aedf1ddd5ba3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:15.271 INFO 26087 --- [prometheus-minute-schedule-1,b1e6570c6130b568,b5726f07272f6621] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:16.433 INFO 26087 --- [ocp-schedules-12,098d6b69b4ef0fbf,92451525ffa8ad69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:45:16.434 INFO 26087 --- [ocp-schedules-27,cae7f836b249c857,e48a06e23852cb0b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:45:16.437 INFO 26087 --- [ocp-schedules-30,539c15d77f9f279c,aefb9d654497a714] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:45:16.438 INFO 26087 --- [ocp-schedules-12,098d6b69b4ef0fbf,92451525ffa8ad69] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:45:16.438 INFO 26087 --- [ocp-schedules-22,0d6301013ec77cf1,15798f269fd38b1b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:45:16.438 INFO 26087 --- [ocp-schedules-22,0d6301013ec77cf1,15798f269fd38b1b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:45:16.438 INFO 26087 --- [ocp-schedules-26,f10834532f27eaae,1af8ea7a9c530446] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-04-08T21:45:16.438+08:00, tenantSize=0 2025-04-08 21:45:16.439 INFO 26087 --- [ocp-schedules-1,9abe9937932e5925,a8ac5a9f22e0bae3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:45:16.439 INFO 26087 --- [ocp-schedules-28,a5ea7339b108958f,de0c978c03baeac0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:45:16.439 INFO 26087 --- [ocp-schedules-1,9abe9937932e5925,a8ac5a9f22e0bae3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:45:16.439 INFO 26087 --- [ocp-schedules-28,a5ea7339b108958f,de0c978c03baeac0] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:45:16.441 INFO 26087 --- [ocp-schedules-27,cae7f836b249c857,e48a06e23852cb0b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:45:16.441 INFO 26087 --- [ocp-schedules-27,2d68c04550be64b5,17726b00d33fb149] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:45:16.442 INFO 26087 --- [ocp-schedules-27,2d68c04550be64b5,17726b00d33fb149] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:45:16.442 INFO 26087 --- [ocp-schedules-3,da65ab409ebf4556,82987c0416284f7a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:45:16.443 INFO 26087 --- [ocp-schedules-3,da65ab409ebf4556,82987c0416284f7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:16.443 INFO 26087 --- [ocp-schedules-13,0fba98fafe7aa17b,2c46c9914d208af0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:45:16.478 INFO 26087 --- [ocp-schedules-2,3e0e5e81a534efc9,14d030563008a614] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2025-04-08 21:45:16.478 INFO 26087 --- [ocp-schedules-2,3e0e5e81a534efc9,14d030563008a614] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-04-08 21:45:16.483 INFO 26087 --- [ocp-schedules-29,44f6b09e63693aee,7eb2d7ced7894fca] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[] 2025-04-08 21:45:16.483 INFO 26087 --- [ocp-schedules-29,44f6b09e63693aee,7eb2d7ced7894fca] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-04-08 21:45:16.493 INFO 26087 --- [ocp-async-10,e7d3e753e50b623b,f576ebbe6c7dd8c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:45:16.505 INFO 26087 --- [ocp-schedules-7,10b2f019d7c2eb72,beb08aa5b9d9c429] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:45:16.505 INFO 26087 --- [ocp-schedules-7,10b2f019d7c2eb72,beb08aa5b9d9c429] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:45:16.542 INFO 26087 --- [ocp-schedules-20,808972b05a7bef14,5cc636383f41f0b8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:45:16.587 INFO 26087 --- [ocp-schedules-15,306e14c2c2821704,9ea00ea801afea9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:45:16.587 INFO 26087 --- [ocp-schedules-15,622ada3f110bf792,b3ebe7579f4936cc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:45:16.588 INFO 26087 --- [ocp-schedules-15,622ada3f110bf792,b3ebe7579f4936cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:16.589 INFO 26087 --- [ocp-schedules-26,1110d29c300cdf13,70399a5e02ecae32] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:45:16.590 INFO 26087 --- [ocp-schedules-14,bb6babeff6288b6b,0bb38c0bd7d9d5ce] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:45:16.590 INFO 26087 --- [ocp-schedules-14,bb6babeff6288b6b,0bb38c0bd7d9d5ce] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:45:16.593 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:45:16.593 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:45:16.601 INFO 26087 --- [ocp-schedules-1,410cf1486d4d7a50,dce8e800892e09eb] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:45:16.601 INFO 26087 --- [ocp-schedules-1,410cf1486d4d7a50,dce8e800892e09eb] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:45:16.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:45:16.623062+08:00, lastProcessTime=2025-04-08T21:45:06.623422+08:00 2025-04-08 21:45:16.625 INFO 26087 --- [ocp-schedules-28,47a4f180ec4ff5f2,3a96d7a2ff6c9961] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:45:16.627 INFO 26087 --- [ocp-schedules-28,47a4f180ec4ff5f2,3a96d7a2ff6c9961] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:45:16.627 INFO 26087 --- [ocp-schedules-28,47a4f180ec4ff5f2,3a96d7a2ff6c9961] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:45:16.637 INFO 26087 --- [ocp-schedules-12,767ddfde8c25ff6f,503fa001e910c946] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:45:16.637+08:00, clusters=[1000003] 2025-04-08 21:45:16.637 INFO 26087 --- [ocp-schedules-13,c1b8b3bab7ef3190,6226491c63ed3b6e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:45:16.638 INFO 26087 --- [ocp-schedules-12,767ddfde8c25ff6f,503fa001e910c946] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:45:16.649 INFO 26087 --- [ocp-async-3,c523b0bdb92de369,5d128b4eb4a49596] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:16.649 WARN 26087 --- [ocp-async-8,b373baf3f865c652,473c65d4b26e5533] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:16.651 INFO 26087 --- [ocp-schedules-25,be48ea78e7311a7d,438f8f05e2f55387] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:45:16.651+08:00, clusters=[1000003] 2025-04-08 21:45:16.652 INFO 26087 --- [ocp-schedules-25,be48ea78e7311a7d,438f8f05e2f55387] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:45:16.654 INFO 26087 --- [ocp-schedules-29,ff24e23d70cd7488,8a2e5118ff908111] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:45:16.654 INFO 26087 --- [ocp-schedules-29,ff24e23d70cd7488,8a2e5118ff908111] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:45:16.658 INFO 26087 --- [ocp-schedules-29,ff24e23d70cd7488,8a2e5118ff908111] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:45:16.658 INFO 26087 --- [ocp-schedules-29,ff24e23d70cd7488,8a2e5118ff908111] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:45:16.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:45:16.623062+08:00 2025-04-08 21:45:16.781 INFO 26087 --- [ocp-schedules-7,8c0a8d03eb0faedb,d8c38978ee4a0315] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:45:16.815 INFO 26087 --- [ocp-schedules-8,8189acfeb30fc100,a93c1fc6088e72a2] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:45:16.816 INFO 26087 --- [ocp-schedules-8,8189acfeb30fc100,a93c1fc6088e72a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:16.827 ERROR 26087 --- [ocp-schedules-8,8189acfeb30fc100,a93c1fc6088e72a2] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:45:16.827 INFO 26087 --- [ocp-schedules-8,8189acfeb30fc100,a93c1fc6088e72a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:45:16.845 INFO 26087 --- [ocp-schedules-11,3ada680095c4f7f0,2c1eb7ccac333461] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:45:16.845 INFO 26087 --- [ocp-schedules-11,3ada680095c4f7f0,2c1eb7ccac333461] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:45:16.845 INFO 26087 --- [ocp-schedules-11,3ada680095c4f7f0,2c1eb7ccac333461] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:45:16.845 INFO 26087 --- [ocp-schedules-11,3ada680095c4f7f0,2c1eb7ccac333461] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:45:16.845 INFO 26087 --- [ocp-schedules-11,3ada680095c4f7f0,2c1eb7ccac333461] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:45:16.845 INFO 26087 --- [ocp-schedules-11,3ada680095c4f7f0,2c1eb7ccac333461] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:45:16.863 INFO 26087 --- [ocp-schedules-21,ce0e3fe9dfbe9c16,248e19eed551932a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:45:16.966 INFO 26087 --- [ocp-schedules-24,293f52f9f07c4923,f8752e584583aed8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:45:16.967 INFO 26087 --- [ocp-schedules-24,293f52f9f07c4923,f8752e584583aed8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:16.983 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:45:16.984 ERROR 26087 --- [ocp-schedules-24,293f52f9f07c4923,f8752e584583aed8] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:45:16.984 INFO 26087 --- [ocp-schedules-24,293f52f9f07c4923,f8752e584583aed8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:45:17.066 INFO 26087 --- [ocp-schedules-10,8a6b8523f2baa1e3,7ecd5f8a7c836cb5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:45:17.629 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:17.630 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:17.680 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-04-08 21:45:17.680 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-04-08 21:45:17.681 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.681 INFO 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:45:17.681 INFO 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:45:17.681 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:45:17.681 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.682 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.682 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.682 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-04-08 21:45:17.683 INFO 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:45:17.684 WARN 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:45:17.685 INFO 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:45:17.686 WARN 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:45:17.689 INFO 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:45:17.691 WARN 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:45:17.691 ERROR 26087 --- [ocp-schedules-2,cdc57be6f0065f77,11ba54fda444380a] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:45:17.695 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.695 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.695 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.695 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-04-08 21:45:17.706 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.706 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.706 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.707 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-04-08 21:45:17.717 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.718 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.718 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.718 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-04-08 21:45:17.729 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.729 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.730 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.730 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-04-08 21:45:17.735 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.735 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.736 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.736 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-04-08 21:45:17.740 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.740 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.740 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.741 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-04-08 21:45:17.744 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.745 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.745 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.745 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-04-08 21:45:17.750 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.750 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.750 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.751 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-04-08 21:45:17.756 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.756 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.756 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.756 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-04-08 21:45:17.760 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.761 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.761 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.761 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-04-08 21:45:17.768 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.768 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.769 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.769 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-04-08 21:45:17.780 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.780 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.780 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.780 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-04-08 21:45:17.784 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.784 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.784 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.784 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-04-08 21:45:17.789 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.789 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.789 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.789 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-04-08 21:45:17.793 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.793 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.793 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.793 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-04-08 21:45:17.796 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.796 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.797 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.797 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-04-08 21:45:17.800 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.800 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.801 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.801 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-04-08 21:45:17.804 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.804 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.805 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.805 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-04-08 21:45:17.809 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.809 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.809 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.810 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-04-08 21:45:17.813 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.814 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.814 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.814 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-04-08 21:45:17.818 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.818 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.818 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.818 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-04-08 21:45:17.821 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.821 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.822 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.822 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-04-08 21:45:17.826 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.826 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.826 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.826 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-04-08 21:45:17.829 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.829 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.829 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.830 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-04-08 21:45:17.833 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.833 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.833 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.833 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-04-08 21:45:17.837 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.837 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.837 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.838 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-04-08 21:45:17.841 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.841 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.841 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.842 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-04-08 21:45:17.845 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.845 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.845 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.846 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-04-08 21:45:17.849 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.849 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.849 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.850 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-04-08 21:45:17.853 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.853 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.854 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.854 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-04-08 21:45:17.858 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.858 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.858 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.858 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-04-08 21:45:17.862 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.862 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.862 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.862 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-04-08 21:45:17.865 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.865 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.866 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.866 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-04-08 21:45:17.871 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.872 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.872 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.872 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-04-08 21:45:17.876 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.876 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.877 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.877 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-04-08 21:45:17.881 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.881 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.881 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.882 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-04-08 21:45:17.886 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.886 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.886 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.887 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-04-08 21:45:17.889 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.890 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.890 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.890 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-04-08 21:45:17.893 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.893 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.893 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.894 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-04-08 21:45:17.896 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.897 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.897 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.897 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-04-08 21:45:17.900 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.900 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.900 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.901 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-04-08 21:45:17.908 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.908 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.909 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.909 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-04-08 21:45:17.925 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.925 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.925 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.926 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-04-08 21:45:17.931 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.931 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.931 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.932 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-04-08 21:45:17.937 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.937 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.937 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.938 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-04-08 21:45:17.942 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.942 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.943 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.943 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-04-08 21:45:17.946 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.946 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.946 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.947 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-04-08 21:45:17.950 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.950 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.951 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.951 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-04-08 21:45:17.954 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-04-08 21:45:17.954 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-04-08 21:45:17.955 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.955 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:45:17.956 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.956 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.957 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.957 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-04-08 21:45:17.960 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.960 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2025-04-08 21:45:17.961 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:17.961 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-04-08 21:45:17.964 INFO 26087 --- [ocp-schedules-22,d96edbcf03fde588,f13966e1703046f4] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-04-08 21:45:18.271 INFO 26087 --- [prometheus-minute-schedule-1,98ccea9341b75909,9f39bcfb6565c4d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:18.391 INFO 26087 --- [ocp-schedules-8,e96579ae15a697ff,1382dddd00b691dd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:45:18.415 INFO 26087 --- [ocp-schedules-8,e96579ae15a697ff,1382dddd00b691dd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:45:18.415 INFO 26087 --- [ocp-schedules-8,e96579ae15a697ff,1382dddd00b691dd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:45:18.649 WARN 26087 --- [ocp-async-7,a6ab76623e81273e,d7f8db7e1e8dc12e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:18.649 INFO 26087 --- [ocp-async-3,f993b7e3833edaae,9f4478159b2680b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:19.945 INFO 26087 --- [ocp-schedules-30,b9f5a9e5baca970e,b6a71dfd8a457af2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:45:19.946 INFO 26087 --- [ocp-schedules-30,b9f5a9e5baca970e,b6a71dfd8a457af2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:45:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:45:20.630 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:20.631 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:20.649 WARN 26087 --- [ocp-async-10,334311ffb53cf64a,3b202c2766cb4f15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:20.649 INFO 26087 --- [ocp-async-4,00b9847af58e68f3,230d8dcc1ef3f75e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:21.271 INFO 26087 --- [prometheus-minute-schedule-1,e5bf0a6da11344b8,e0d63b29ebb7cdfa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:22.650 WARN 26087 --- [ocp-async-9,51a0b85a872f3521,bc90e913fbd50406] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:22.650 INFO 26087 --- [ocp-async-4,6689c8419de47061,5b72a1a0a141984f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:23.172 INFO 26087 --- [ocp-schedules-24,65b7f0f7fe40db3e,2a4f287e07f2545a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:45:23.631 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:23.631 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:24.271 INFO 26087 --- [prometheus-minute-schedule-1,d6e3041444457856,68db5ea5fd050534] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:24.650 INFO 26087 --- [ocp-async-6,43e400999305f035,7694ef2712b81cfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:24.650 WARN 26087 --- [ocp-async-1,04b31232313e6386,d94034599c0a287a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:26.442 INFO 26087 --- [ocp-schedules-27,2d68c04550be64b5,17726b00d33fb149] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:45:26.443 INFO 26087 --- [ocp-schedules-3,da65ab409ebf4556,82987c0416284f7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:45:26.493 INFO 26087 --- [ocp-async-6,97478356f415401e,0b7022d186cbdc52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:45:26.510 INFO 26087 --- [ocp-schedules-24,c2acd179065ed06d,dad229dc27a5f0f9] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:45:26.510 INFO 26087 --- [ocp-schedules-24,c2acd179065ed06d,dad229dc27a5f0f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:26.511 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:45:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:45:26.622742+08:00, lastProcessTime=2025-04-08T21:45:16.623062+08:00 2025-04-08 21:45:26.631 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:26.632 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:26.650 WARN 26087 --- [ocp-async-3,c5df9c00be8c2113,18f3502d53e76ff6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:26.650 INFO 26087 --- [ocp-async-9,6e2f562d1eeb01cb,68cb61f3545f52c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:26.719 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:45:26.622742+08:00 2025-04-08 21:45:26.782 INFO 26087 --- [ocp-schedules-26,01509c91c6430c7f,20b1da49579aec5b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:45:27.271 INFO 26087 --- [prometheus-minute-schedule-1,37e62ce1c785a5f2,44d66929834b0b7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:28.650 WARN 26087 --- [ocp-async-9,bfc98db9db90bce6,0148ec59fab6d2bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:28.650 INFO 26087 --- [ocp-async-1,a21d23ba9be3778b,92bf2bb37af29b4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:29.632 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:29.633 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:29.996 INFO 26087 --- [ocp-schedules-3,1c76ef13d187711b,26c41f3ff20b56ae] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:45:29.997 INFO 26087 --- [ocp-schedules-3,1c76ef13d187711b,26c41f3ff20b56ae] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:45:30.271 INFO 26087 --- [prometheus-minute-schedule-1,f8369ac5ec751e11,935e616e1aebb70a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:30.650 WARN 26087 --- [ocp-async-6,8726fe931342e6f4,468419f8264ae79a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:30.650 INFO 26087 --- [ocp-async-7,eb97e7597c9fcdfd,9c426f4db819881c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:32.633 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:32.633 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:32.650 WARN 26087 --- [ocp-async-5,1f957a5a2b5fbccd,80f31a9d3ab6670b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:32.650 INFO 26087 --- [ocp-async-7,5e5ea5d485fc127f,6665d7e1e2a48652] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:33.183 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:45:33.183 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:45:33.186 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:45:33.186 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:45:33.186 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:45:33.186 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:45:33.189 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:45:33.189 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:45:33.189 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:45:33.189 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:45:33.197 INFO 26087 --- [ocp-schedules-21,61c0f0a79ee5f64d,13278f061485cad3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:45:33.229 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:45:33.235 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:45:33.235 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:45:33.236 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:45:33.236 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:45:33.236 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:45:33.240 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:45:33.271 INFO 26087 --- [prometheus-minute-schedule-1,d3f394a3288c2e8b,9c3d8e101ff00843] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:33.544 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:45:33.884 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.694837696S 2025-04-08 21:45:33.884 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:45:33.939 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:45:33.939 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:45:33.963 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:45:33.999 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:45:33.999 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:45:33.999 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:45:33.999 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:45:33.999 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:45:33.999 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:45:34.007 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:45:34.106 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] o.s.boot.SpringApplication : Started application in 0.919 seconds (JVM running for 2639.562) 2025-04-08 21:45:34.106 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.920038282S 2025-04-08 21:45:34.382 INFO 26087 --- [ocp-schedules-20,4cb4a092379cda98,cf45d646276261f4] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:45:34.651 WARN 26087 --- [ocp-async-2,32c040d6c72b7055,7effc15fb8d16e02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:34.651 INFO 26087 --- [ocp-async-8,1c671c48ba59783a,a1df33338a988d59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:45:34.912 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:45:34.912 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:45:34.913 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:45:34.917 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:45:34.917 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 21:45:35.633 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:35.634 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:36.271 INFO 26087 --- [prometheus-minute-schedule-1,6473a5a205ffb2ee,4fc4d24032566fa4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:36.493 INFO 26087 --- [ocp-async-9,718906f58582eee1,108b9c24eab63b99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:45:36.511 INFO 26087 --- [ocp-schedules-24,c2acd179065ed06d,dad229dc27a5f0f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:45:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:45:36.623115+08:00, lastProcessTime=2025-04-08T21:45:26.622742+08:00 2025-04-08 21:45:36.651 WARN 26087 --- [ocp-async-10,a14e5a4d93a8c920,b09c78b818c5dba3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:36.651 INFO 26087 --- [ocp-async-1,26157132796c3b7b,f5f3616fdf5b0ff7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:36.724 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:45:36.623115+08:00 2025-04-08 21:45:36.783 INFO 26087 --- [ocp-schedules-11,44307f7a5ac9b398,f88071baa69a776d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:45:38.416 INFO 26087 --- [ocp-schedules-6,e1ce67d1f172cf45,fb7948aa8fc4759a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:45:38.423 INFO 26087 --- [ocp-schedules-6,e1ce67d1f172cf45,fb7948aa8fc4759a] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:45:38.423 INFO 26087 --- [ocp-schedules-6,e1ce67d1f172cf45,fb7948aa8fc4759a] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:45:38.634 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:38.634 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:38.651 WARN 26087 --- [ocp-async-1,4070d09bda27f6be,72ae9f4af5b81e53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:38.651 INFO 26087 --- [ocp-async-3,6c7195cc673bc758,1c7d2c112b948402] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:39.271 INFO 26087 --- [prometheus-minute-schedule-1,e767518c22647eb3,ae6a94c1b06dd903] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:40.001 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:45:40.001 INFO 26087 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:45:40.002 INFO 26087 --- [ocp-schedules-12,0258bb81b278dfb0,d900ac8352528c89] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:45:40.003 INFO 26087 --- [ocp-schedules-12,0258bb81b278dfb0,d900ac8352528c89] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:45:40.651 WARN 26087 --- [ocp-async-2,203cfc1a51fd5c2b,3e6e1fdf6d0e8e45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:40.651 INFO 26087 --- [ocp-async-4,cd82b375c46c4522,1d23dc6dce7592ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:41.635 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:41.635 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:42.271 INFO 26087 --- [prometheus-minute-schedule-1,8ae717057b66ca8e,63aa6f52ff52ed51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:42.651 WARN 26087 --- [ocp-async-8,578ce31d19954a17,b307fb29e8fe4c21] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:42.651 INFO 26087 --- [ocp-async-10,7659323a185c8087,9e457d8552c42c7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:43.234 INFO 26087 --- [ocp-schedules-4,ebf2d6f010c0db0c,e3f0edd4fefc2ef0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:45:44.635 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:44.636 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:44.651 WARN 26087 --- [ocp-async-7,665017d5ccc1b0fb,cd1e202181c9a40a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:44.651 INFO 26087 --- [ocp-async-1,96fb40777235a0f7,a60c4724db223a87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:45.271 INFO 26087 --- [prometheus-minute-schedule-1,f762a977318a97e2,658202cd2a3d4694] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:46.437 INFO 26087 --- [ocp-schedules-4,54e6c05e1624193c,7d454eb4acb9c018] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:45:46.438 INFO 26087 --- [ocp-schedules-3,290685212bbf89d7,ea840069694167d9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:45:46.438 INFO 26087 --- [ocp-schedules-3,290685212bbf89d7,ea840069694167d9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:45:46.439 INFO 26087 --- [ocp-schedules-20,ab9e8c8b8ff64fb1,e345acc7cef3ba15] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:45:46.439 INFO 26087 --- [ocp-schedules-20,ab9e8c8b8ff64fb1,e345acc7cef3ba15] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:45:46.440 INFO 26087 --- [ocp-schedules-7,29678f536ea33f50,8002bf2c89828d5f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:45:46.440 INFO 26087 --- [ocp-schedules-7,29678f536ea33f50,8002bf2c89828d5f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:45:46.441 INFO 26087 --- [ocp-schedules-12,33a3b01fddf3beee,f1ec9fbca66f99ee] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:45:46.441 INFO 26087 --- [ocp-schedules-24,5bd0c8335151fb31,b5b367c9d7d6065a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:45:46.441 INFO 26087 --- [ocp-schedules-12,33a3b01fddf3beee,f1ec9fbca66f99ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:46.441 INFO 26087 --- [ocp-schedules-24,5bd0c8335151fb31,b5b367c9d7d6065a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:45:46.458 INFO 26087 --- [ocp-schedules-10,fb94e39b748724d6,f601ceb2f42b4c90] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:45:46.494 INFO 26087 --- [ocp-async-5,60fbeb38c5f94e2e,a9a1cc73a6fc82b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:45:46.591 INFO 26087 --- [ocp-schedules-13,760bd6fd9c960323,cb704b68b2f6d096] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:45:46.593 INFO 26087 --- [ocp-schedules-15,622ada3f110bf792,b3ebe7579f4936cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:45:46.593 INFO 26087 --- [ocp-schedules-15,fca261c996e81c42,e0ac18867e6803a4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:45:46.594 INFO 26087 --- [ocp-schedules-15,fca261c996e81c42,e0ac18867e6803a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:45:46.598 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:45:46.598 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:45:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:45:46.622753+08:00, lastProcessTime=2025-04-08T21:45:36.623115+08:00 2025-04-08 21:45:46.651 WARN 26087 --- [ocp-async-9,2b2e0e9aefae3048,c24b83fe9c205508] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:46.651 INFO 26087 --- [ocp-async-2,e366475d890514fc,8b78a9c5c74f98c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:46.755 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:45:46.622753+08:00 2025-04-08 21:45:46.785 INFO 26087 --- [ocp-schedules-25,5cd869c309be17ed,64ef735857a17bd8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:45:47.636 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:47.637 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:48.271 INFO 26087 --- [prometheus-minute-schedule-1,c400ee71017d2913,bb5aaf0de4e0bab3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:48.652 WARN 26087 --- [ocp-async-2,2de62fd0c219be46,a245939c2925e80f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:48.652 INFO 26087 --- [ocp-async-6,0175813886c42684,2bdf5898d9b69a78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:50.037 INFO 26087 --- [ocp-schedules-4,0870f96e42592b0d,2372e676ea493dc6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:45:50.038 INFO 26087 --- [ocp-schedules-4,0870f96e42592b0d,2372e676ea493dc6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:45:50.637 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:50.638 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:50.652 WARN 26087 --- [ocp-async-5,c9cc65c328adeb0b,c94ebbb4fa4f82a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:50.652 INFO 26087 --- [ocp-async-7,361ab3cdb135c88b,27da56a1d28660b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:51.271 INFO 26087 --- [prometheus-minute-schedule-1,93cb13afb689b0fc,acfd6cc6dd6b8176] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:52.652 WARN 26087 --- [ocp-async-4,179f62cd2674e184,5f15c94eb4dc78c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:52.652 INFO 26087 --- [ocp-async-9,dad1e492ed095721,30843374d5b74e5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:53.258 INFO 26087 --- [ocp-schedules-16,303d7eafe32b28be,e9ff935dc8436590] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:45:53.625 INFO 26087 --- [http-nio-0.0.0.0-8080-exec-3,ddca80ee4402c8a9,7be28e2f60d4a8c4] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sgkydb&version=2&ObClusterId=1744016068, client=192.168.1.62, traceId=ddca80ee4402c8a9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744119953621754,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-04-08 21:45:53.626 WARN 26087 --- [http-nio-0.0.0.0-8080-exec-3,ddca80ee4402c8a9,7be28e2f60d4a8c4] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /services client=192.168.1.62, traceId=ddca80ee4402c8a9, args=sgkydb,1744016068,2,{"ObRegion":"sgkydb","ObCluster":"sgkydb","ObRegionId":1744016068,"ObClusterId":1744016068,"Type":"PRIMARY","timestamp":1744119953621754,"RsList":[{"address":"192.168.1.62:2882","role":"LEADER","sql_port":2881},{"address":"192.168.1.64:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},, duration=3 ms, errorMsg=[OCP BadRequestException]: status=400 BAD_REQUEST, errorCode=CONFIGURL_OB_CLUSTER_NOT_EXIST, args=] 2025-04-08 21:45:53.638 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:53.638 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:54.271 INFO 26087 --- [prometheus-minute-schedule-1,c76628de8c6080e5,86ef90ed848b5d6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:54.652 WARN 26087 --- [ocp-async-8,c8467ac6e76def39,200cb25224355db0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:54.652 INFO 26087 --- [ocp-async-2,baf2b3428ffa86eb,62ee9cf911070ca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:56.442 INFO 26087 --- [ocp-schedules-12,33a3b01fddf3beee,f1ec9fbca66f99ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:45:56.442 INFO 26087 --- [ocp-schedules-24,5bd0c8335151fb31,b5b367c9d7d6065a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:45:56.494 INFO 26087 --- [ocp-async-10,fd8273d11a7677c6,93e26adad5029290] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:45:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:45:56.624494+08:00, lastProcessTime=2025-04-08T21:45:46.622753+08:00 2025-04-08 21:45:56.638 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:56.639 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:45:56.652 WARN 26087 --- [ocp-async-5,27a6566279b699d6,01a6b687231bfb59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:56.652 INFO 26087 --- [ocp-async-1,171a7936a94add5e,076e29ea2d69a197] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:56.736 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:45:56.624494+08:00 2025-04-08 21:45:56.786 INFO 26087 --- [ocp-schedules-11,234258ead2d09697,ac4c0537f0c728d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:45:57.271 INFO 26087 --- [prometheus-minute-schedule-1,b02c013f1647d252,00949337d6061ec0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744119960, interval=60 2025-04-08 21:45:58.424 INFO 26087 --- [ocp-schedules-22,4b453415e5a47fce,fa6a1985325925e3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:45:58.429 INFO 26087 --- [ocp-schedules-22,4b453415e5a47fce,fa6a1985325925e3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:45:58.429 INFO 26087 --- [ocp-schedules-22,4b453415e5a47fce,fa6a1985325925e3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:45:58.652 WARN 26087 --- [ocp-async-3,efb31c3b41c0e81e,d46498b87ed1912d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:45:58.652 INFO 26087 --- [ocp-async-4,82d01cdb8c62c6be,208627c66f0cb24c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:45:59.639 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:45:59.639 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:00.063 INFO 26087 --- [ocp-schedules-12,561e1c6fa02501f9,c5d3d52a099e1791] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:46:00.064 INFO 26087 --- [ocp-schedules-12,561e1c6fa02501f9,c5d3d52a099e1791] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:46:00.271 INFO 26087 --- [prometheus-minute-schedule-1,d9b8faa5ae9a38b0,76797d7d25d70fb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:00.652 WARN 26087 --- [ocp-async-2,f001594fc9c35fec,7a19bd7b4cc1b0be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:00.653 INFO 26087 --- [ocp-async-8,3b6a952cb2e04bc7,5ac755f9966ef303] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:02.639 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:02.640 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:02.653 WARN 26087 --- [ocp-async-5,ba33e4f6a663a1c9,fc068973ba9e628b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:02.653 INFO 26087 --- [ocp-async-8,53789117531bb896,c4749b9784851019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:03.271 INFO 26087 --- [prometheus-minute-schedule-1,cd069473b6a5c97f,a3523991a4b74d49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:03.277 INFO 26087 --- [ocp-schedules-8,f4257021b82e3f41,979b7e56b167805c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:46:04.653 INFO 26087 --- [ocp-async-8,7d93b95ce7faef88,5d1dc22fd17dac90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:04.653 WARN 26087 --- [ocp-async-3,37ef8053eb4421c3,e7d56b0c3cd05a08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:05.640 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:05.640 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:06.271 INFO 26087 --- [prometheus-minute-schedule-1,0a6dc15b69a9c0f1,da7865377aafc456] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:06.272 INFO 26087 --- [prometheus-cache-monitor1,d18b67fe850b4d1f,f3298e79d912fd0c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:46:06.272 INFO 26087 --- [prometheus-cache-monitor1,d18b67fe850b4d1f,f3298e79d912fd0c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:46:06.494 INFO 26087 --- [ocp-async-1,2bacd79ccbc369f6,9f761a3ecc2c8dee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:46:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:46:06.622887+08:00, lastProcessTime=2025-04-08T21:45:56.624494+08:00 2025-04-08 21:46:06.653 WARN 26087 --- [ocp-async-4,fd8f65b806b04b36,451b5a2c87253d58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:06.653 INFO 26087 --- [ocp-async-2,c6dfc68428bb99b0,7e76e21ff4f16f9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:06.745 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:46:06.622887+08:00 2025-04-08 21:46:06.788 INFO 26087 --- [ocp-schedules-3,e71671384f5206c6,2c3a30592a5170dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:46:06.882 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:42:06.859, end=2025-04-08T21:44:06.88} 2025-04-08 21:46:06.887 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:46:06.887 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:46:08.640 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:08.641 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:08.653 WARN 26087 --- [ocp-async-5,d1add56aab1fd5d4,d99998b7a984544b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:08.653 INFO 26087 --- [ocp-async-2,40b6411411493f16,77f81d25f3292fd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:09.271 INFO 26087 --- [prometheus-minute-schedule-1,a76ca240f2dbd941,60519497bb286e6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:10.001 INFO 26087 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:46:10.002 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:46:10.100 INFO 26087 --- [ocp-schedules-23,563796bd27362337,8dee78253cf0a715] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:46:10.101 INFO 26087 --- [ocp-schedules-23,563796bd27362337,8dee78253cf0a715] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:46:10.520 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:46:10.653 WARN 26087 --- [ocp-async-3,eead5bdba98827cb,60b266a8a7117f41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:10.653 INFO 26087 --- [ocp-async-9,08eb0f34f7e3b2a3,c1193f5e8b5ee68e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:11.641 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:11.642 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:12.271 INFO 26087 --- [prometheus-minute-schedule-1,cc079f0701f55612,c91433d9512b4c6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:12.502 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119900, nodeCount=188, affectRows=188 2025-04-08 21:46:12.503 INFO 26087 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119900, nodeCount=187, validCount=187 2025-04-08 21:46:12.534 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119900, nodeCount=192, affectRows=192 2025-04-08 21:46:12.535 INFO 26087 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119900, nodeCount=190, validCount=190 2025-04-08 21:46:12.653 WARN 26087 --- [ocp-async-7,7efc74eae2680c9d,3b76b5d267146aff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:12.653 INFO 26087 --- [ocp-async-9,1aca269c1eb5ae2c,d40b0da2202c9145] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:13.303 INFO 26087 --- [ocp-schedules-19,2b1dcecd64eb5e29,7c5ecb3d25f13a3e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:46:14.642 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:14.642 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:14.653 WARN 26087 --- [ocp-async-9,209ad63036025045,b9688fb5a10fa361] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:14.653 INFO 26087 --- [ocp-async-6,489cf346498797a8,dffef9b02bdc3516] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:15.271 INFO 26087 --- [prometheus-minute-schedule-1,fd98b0d7f0faa17b,399254420cf8eead] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:16.433 INFO 26087 --- [ocp-schedules-5,c50a8fd46f466c4c,395e1844d372ae16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:46:16.434 INFO 26087 --- [ocp-schedules-29,bf44194962b26911,7d557124a4736239] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:46:16.437 INFO 26087 --- [ocp-schedules-25,7c6205201f4377fb,7f3cbe40f379d82c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:46:16.437 INFO 26087 --- [ocp-schedules-25,7c6205201f4377fb,7f3cbe40f379d82c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:46:16.437 INFO 26087 --- [ocp-schedules-25,b436b4a6aebc9d8f,24a854d2a55cd277] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:46:16.439 INFO 26087 --- [ocp-schedules-1,b9a7d53163334146,8c8d897e7674247e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:46:16.439 INFO 26087 --- [ocp-schedules-1,b9a7d53163334146,8c8d897e7674247e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:46:16.439 INFO 26087 --- [ocp-schedules-21,e1a2b80a1afeaff5,1c4cfdb5d92a9cba] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:46:16.439 INFO 26087 --- [ocp-schedules-21,e1a2b80a1afeaff5,1c4cfdb5d92a9cba] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:46:16.440 INFO 26087 --- [ocp-schedules-5,c50a8fd46f466c4c,395e1844d372ae16] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:46:16.440 INFO 26087 --- [ocp-schedules-29,bf44194962b26911,7d557124a4736239] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:46:16.441 INFO 26087 --- [ocp-schedules-19,faf3d47d86c0fa27,2b77011b4861d6fd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:46:16.441 INFO 26087 --- [ocp-schedules-8,a7c24192e3edd8d4,147170a727feffc1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:46:16.441 INFO 26087 --- [ocp-schedules-19,faf3d47d86c0fa27,2b77011b4861d6fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:16.441 INFO 26087 --- [ocp-schedules-8,a7c24192e3edd8d4,147170a727feffc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:46:16.446 INFO 26087 --- [ocp-schedules-27,3323887fd81716f1,69e68c1979b47adc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:46:16.484 INFO 26087 --- [ocp-schedules-7,26076a85111df27a,5695a0d1a3670eec] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:46:16.484 INFO 26087 --- [ocp-schedules-7,26076a85111df27a,5695a0d1a3670eec] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:46:16.494 INFO 26087 --- [ocp-async-3,c3462ba12caa2f73,2f9d13704748b846] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:46:16.544 INFO 26087 --- [ocp-schedules-10,7e40e4ca3a7b2902,91b59922876baabb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:46:16.592 INFO 26087 --- [ocp-schedules-17,4ca32df942fd90f7,8ab31176478d6127] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:46:16.592 INFO 26087 --- [ocp-schedules-16,88b6c65f1fa31976,c364a926d234a098] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:46:16.592 INFO 26087 --- [ocp-schedules-16,88b6c65f1fa31976,c364a926d234a098] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:46:16.599 INFO 26087 --- [ocp-schedules-15,fca261c996e81c42,e0ac18867e6803a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:46:16.599 INFO 26087 --- [ocp-schedules-15,fecb88b3ffd8819a,bd1615848d2f190e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:46:16.600 INFO 26087 --- [ocp-schedules-15,fecb88b3ffd8819a,bd1615848d2f190e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:16.603 INFO 26087 --- [ocp-schedules-18,1a43c963f90de8ac,5703822a392f27db] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:46:16.603 INFO 26087 --- [ocp-schedules-18,1a43c963f90de8ac,5703822a392f27db] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:46:16.604 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:46:16.604 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:46:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:46:16.624119+08:00, lastProcessTime=2025-04-08T21:46:06.622887+08:00 2025-04-08 21:46:16.627 INFO 26087 --- [ocp-schedules-25,969d86e0bd99072d,dc5911cd6aeb7310] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:46:16.628 INFO 26087 --- [ocp-schedules-25,969d86e0bd99072d,dc5911cd6aeb7310] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:46:16.628 INFO 26087 --- [ocp-schedules-25,969d86e0bd99072d,dc5911cd6aeb7310] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:46:16.643 INFO 26087 --- [ocp-schedules-1,5bd453e51bc6e902,f9bb312bc184edb6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:46:16.643 INFO 26087 --- [ocp-schedules-21,560cf1840f18d4d9,160a6fb0f3c96aab] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:46:16.643+08:00, clusters=[1000003] 2025-04-08 21:46:16.643 INFO 26087 --- [ocp-schedules-21,560cf1840f18d4d9,160a6fb0f3c96aab] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:46:16.654 WARN 26087 --- [ocp-async-8,a9f5450e0fef556a,1b748162e1ce547c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:16.654 INFO 26087 --- [ocp-async-7,fc6b0de12bbbbd37,4b06d8e4d93a945d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:16.657 INFO 26087 --- [ocp-schedules-29,88943414426491d3,2a41d3cf4fbca58a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:46:16.657+08:00, clusters=[1000003] 2025-04-08 21:46:16.658 INFO 26087 --- [ocp-schedules-3,f391b743f72cfd45,78e1fb68a4f294a2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:46:16.658 INFO 26087 --- [ocp-schedules-3,f391b743f72cfd45,78e1fb68a4f294a2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:46:16.658 INFO 26087 --- [ocp-schedules-29,88943414426491d3,2a41d3cf4fbca58a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:46:16.662 INFO 26087 --- [ocp-schedules-3,f391b743f72cfd45,78e1fb68a4f294a2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:46:16.662 INFO 26087 --- [ocp-schedules-3,f391b743f72cfd45,78e1fb68a4f294a2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:46:16.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:46:16.624119+08:00 2025-04-08 21:46:16.789 INFO 26087 --- [ocp-schedules-12,617536040daa42bd,71fbbd7119ff4e37] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:46:16.846 INFO 26087 --- [ocp-schedules-6,e63a6051b1baa81c,83de2b96885a2dcc] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:46:16.846 INFO 26087 --- [ocp-schedules-6,e63a6051b1baa81c,83de2b96885a2dcc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:46:16.849 INFO 26087 --- [ocp-schedules-6,e63a6051b1baa81c,83de2b96885a2dcc] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:46:16.849 INFO 26087 --- [ocp-schedules-6,e63a6051b1baa81c,83de2b96885a2dcc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:46:16.851 INFO 26087 --- [ocp-schedules-6,e63a6051b1baa81c,83de2b96885a2dcc] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:46:16.851 INFO 26087 --- [ocp-schedules-6,e63a6051b1baa81c,83de2b96885a2dcc] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:46:16.863 INFO 26087 --- [ocp-schedules-20,9d16936441a53cb4,1a9edde173261ab5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:46:16.864 INFO 26087 --- [ocp-schedules-20,9d16936441a53cb4,1a9edde173261ab5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:16.876 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:46:16.876 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:46:16.985 INFO 26087 --- [ocp-schedules-2,4b63046d3d5ddbb7,5c8a4417c5cbf455] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:46:16.986 INFO 26087 --- [ocp-schedules-2,4b63046d3d5ddbb7,5c8a4417c5cbf455] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:17.002 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:46:17.002 ERROR 26087 --- [ocp-schedules-2,4b63046d3d5ddbb7,5c8a4417c5cbf455] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:46:17.003 INFO 26087 --- [ocp-schedules-2,4b63046d3d5ddbb7,5c8a4417c5cbf455] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:46:17.087 INFO 26087 --- [ocp-schedules-10,04a1d0731b32b67e,700854afbd595356] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:46:17.642 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:17.643 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:17.694 INFO 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:46:17.694 INFO 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:46:17.697 INFO 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:46:17.698 WARN 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:46:17.698 INFO 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:46:17.699 WARN 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:46:17.703 INFO 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:46:17.704 WARN 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:46:17.711 ERROR 26087 --- [ocp-schedules-25,d1d1f868f45760b4,93ca0bf660d9d8b7] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:46:18.271 INFO 26087 --- [prometheus-minute-schedule-1,e87d3f8ab94af43e,2cfb7985e987c022] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:18.430 INFO 26087 --- [ocp-schedules-23,2d1236b54dc01a06,54137552759ee817] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:46:18.446 INFO 26087 --- [ocp-schedules-23,2d1236b54dc01a06,54137552759ee817] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:46:18.446 INFO 26087 --- [ocp-schedules-23,2d1236b54dc01a06,54137552759ee817] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:46:18.654 WARN 26087 --- [ocp-async-9,e62feadd7ee82ac9,3c5bd910697444cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:18.654 INFO 26087 --- [ocp-async-7,7064167de795450b,abd0229cbc88b09d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:20.179 INFO 26087 --- [ocp-schedules-10,d6b7a5adad4a1e72,0e0b31f054269250] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:46:20.180 INFO 26087 --- [ocp-schedules-10,d6b7a5adad4a1e72,0e0b31f054269250] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:46:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:46:20.643 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:20.643 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:20.654 WARN 26087 --- [ocp-async-7,0348b0d4e89d52bb,4c64052620b502d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:20.654 INFO 26087 --- [ocp-async-6,4dcb6b3f57bab2b0,f1659db252f8bb30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:21.271 INFO 26087 --- [prometheus-minute-schedule-1,672be1d379e40411,fb4b47eab47459d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:22.654 WARN 26087 --- [ocp-async-8,6c616b3b4374efbb,27d8dacf9d9e54d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:22.654 INFO 26087 --- [ocp-async-6,312534ca20c6fcc9,0bb62e27b6c3e516] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:23.323 INFO 26087 --- [ocp-schedules-2,0547d97d1c02d417,31b26d5401a530fb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:46:23.644 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:23.644 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:24.271 INFO 26087 --- [prometheus-minute-schedule-1,4f9c6cbe63f81722,301c3aecf2504f95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:24.654 INFO 26087 --- [ocp-async-4,48fbd94c2dc1c375,d9df5c60bf303721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:24.654 WARN 26087 --- [ocp-async-5,93873a48b71082d7,7cd022c5f29c3f9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:26.442 INFO 26087 --- [ocp-schedules-8,a7c24192e3edd8d4,147170a727feffc1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:46:26.442 INFO 26087 --- [ocp-schedules-19,faf3d47d86c0fa27,2b77011b4861d6fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:46:26.494 INFO 26087 --- [ocp-async-2,25c0ea030a198197,744baf1dc06624e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:46:26.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:46:26.624103+08:00, lastProcessTime=2025-04-08T21:46:16.624119+08:00 2025-04-08 21:46:26.644 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:26.645 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:26.654 WARN 26087 --- [ocp-async-1,0f4a65e3536312b3,5ab8b3eb76bbbf0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:26.654 INFO 26087 --- [ocp-async-3,1c5cea3ff368a99f,6cabe7a5049a24af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:26.758 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:46:26.624103+08:00 2025-04-08 21:46:26.790 INFO 26087 --- [ocp-schedules-1,1b7d2d08ec06daa4,23b66154f669e86d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:46:27.271 INFO 26087 --- [prometheus-minute-schedule-1,3a3dad27ce4bb785,3ef4004194cf84c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:28.654 INFO 26087 --- [ocp-async-10,a92d45e61fe2042c,a43b7f94bad7cab4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:28.655 WARN 26087 --- [ocp-async-6,85c5ddd85579b9fa,f5037f482a7f62a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:29.645 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:29.645 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:30.188 INFO 26087 --- [ocp-schedules-12,d8fb64339529ef77,a5e3d94dee9bcf4c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:46:30.189 INFO 26087 --- [ocp-schedules-12,d8fb64339529ef77,a5e3d94dee9bcf4c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:46:30.271 INFO 26087 --- [prometheus-minute-schedule-1,b45bd9ebafdf2dab,3c2ebcbfab890903] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:30.655 WARN 26087 --- [ocp-async-4,d1732f4605f28296,fb84d518c7ec341f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:30.655 INFO 26087 --- [ocp-async-9,d91fd498158d17d2,f1bcd0f9effd7264] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:32.645 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:32.646 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:32.655 WARN 26087 --- [ocp-async-7,9b4923eed0095b5c,f8eef64f8cdca6a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:32.655 INFO 26087 --- [ocp-async-8,c38dcf1dca6de53d,49f0f0dfb8acc575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:33.271 INFO 26087 --- [prometheus-minute-schedule-1,f43d88b656e2a35f,a3e5575ee0bd0e4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:33.346 INFO 26087 --- [ocp-schedules-3,4579577e612cab75,33aab4a8a0401340] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:46:34.655 WARN 26087 --- [ocp-async-5,0c6eb491b4b21baa,1ca9a1338c9763e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:34.655 INFO 26087 --- [ocp-async-3,e800d101a824827c,9c352e838304d622] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:35.646 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:35.647 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:36.271 INFO 26087 --- [prometheus-minute-schedule-1,a074827bcedf8fa9,f8dd5a87fabc0f85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:36.494 INFO 26087 --- [ocp-async-6,b1b0c6a2a4d4f2bc,1fb15b17558a29ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:46:36.511 INFO 26087 --- [ocp-schedules-16,661ad3770540ad1a,d3c064f0d9047d70] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:46:36.512 INFO 26087 --- [ocp-schedules-16,661ad3770540ad1a,d3c064f0d9047d70] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:36.512 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:46:36.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:46:36.623497+08:00, lastProcessTime=2025-04-08T21:46:26.624103+08:00 2025-04-08 21:46:36.655 WARN 26087 --- [ocp-async-2,09742133eadf39e2,2edccbdfe8121bcf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:36.655 INFO 26087 --- [ocp-async-9,6efa19b594b588cc,cf0b3b350586e0b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:36.711 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:46:36.623497+08:00 2025-04-08 21:46:36.791 INFO 26087 --- [ocp-schedules-8,7f58da1599c8414f,9d1f387f06c4c844] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:46:38.447 INFO 26087 --- [ocp-schedules-25,0e1b6308b9556d54,e03692f3066bcb7e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:46:38.470 INFO 26087 --- [ocp-schedules-25,0e1b6308b9556d54,e03692f3066bcb7e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:46:38.470 INFO 26087 --- [ocp-schedules-25,0e1b6308b9556d54,e03692f3066bcb7e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:46:38.647 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:38.647 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:38.655 WARN 26087 --- [ocp-async-7,2d4e6f7897f016f5,9694c44393565c0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:38.655 INFO 26087 --- [ocp-async-8,8f145a135ef40cbc,53c9ba914441d54b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:39.271 INFO 26087 --- [prometheus-minute-schedule-1,b906ce2a8cfaa9fe,b1e8ebf810d104ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:40.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:46:40.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:46:40.200 INFO 26087 --- [ocp-schedules-28,0849f9427bdc75ed,9ce1c16e427cf262] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:46:40.201 INFO 26087 --- [ocp-schedules-28,0849f9427bdc75ed,9ce1c16e427cf262] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:46:40.655 WARN 26087 --- [ocp-async-5,e878bf2ed2da5dca,739f42ec17c02c86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:40.655 INFO 26087 --- [ocp-async-6,5cef7078af940d3a,a8ec4e50b32b740e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:41.648 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:41.648 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:42.271 INFO 26087 --- [prometheus-minute-schedule-1,3929a1ee6afb1397,5cd14558f19699a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:42.655 WARN 26087 --- [ocp-async-4,a80fd00f91b758e3,5b03fffbf3540e1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:42.656 INFO 26087 --- [ocp-async-2,9e4ca5cb7dad77b4,296df266184ed9db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:43.370 INFO 26087 --- [ocp-schedules-21,4cf5dd5b7ed40a01,bb6856cd97a8f613] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:46:44.648 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:44.649 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:44.656 WARN 26087 --- [ocp-async-1,18c8a60d94035706,ef62bd43dc27f79f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:44.656 INFO 26087 --- [ocp-async-9,17bfe43fe0840d1b,8a8b2e798a168952] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:45.271 INFO 26087 --- [prometheus-minute-schedule-1,cc5a94eaeeaf2aa9,32d6f7f9e4cede83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:46.437 INFO 26087 --- [ocp-schedules-2,8a07370dbef83c2b,9178fb88dbe085e4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:46:46.437 INFO 26087 --- [ocp-schedules-24,4b611a1d03ea9e30,759a0010276a21e4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:46:46.437 INFO 26087 --- [ocp-schedules-24,4b611a1d03ea9e30,759a0010276a21e4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:46:46.440 INFO 26087 --- [ocp-schedules-23,405ba333ef3a145d,9cf6a95a197eb78f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:46:46.440 INFO 26087 --- [ocp-schedules-23,405ba333ef3a145d,9cf6a95a197eb78f] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:46:46.440 INFO 26087 --- [ocp-schedules-13,87a3501afe2c06de,5cad009f1db4e8c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:46:46.440 INFO 26087 --- [ocp-schedules-13,87a3501afe2c06de,5cad009f1db4e8c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:46:46.441 INFO 26087 --- [ocp-schedules-1,db93aa9328701757,2a0e10ca405a0ed6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:46:46.441 INFO 26087 --- [ocp-schedules-25,c28634f835d95c1f,f3007197e8efff5a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:46:46.441 INFO 26087 --- [ocp-schedules-1,db93aa9328701757,2a0e10ca405a0ed6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:46:46.442 INFO 26087 --- [ocp-schedules-25,c28634f835d95c1f,f3007197e8efff5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:46.470 INFO 26087 --- [ocp-schedules-12,2a4f7798a23c40f1,43a38265f8db5a9a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:46:46.494 INFO 26087 --- [ocp-async-7,0bf5372f4edb4213,ba819146af605c22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:46:46.512 INFO 26087 --- [ocp-schedules-16,661ad3770540ad1a,d3c064f0d9047d70] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:46:46.593 INFO 26087 --- [ocp-schedules-4,903c600273e3d46d,dd7991740b7ebfdc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:46:46.605 INFO 26087 --- [ocp-schedules-15,fecb88b3ffd8819a,bd1615848d2f190e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:46:46.605 INFO 26087 --- [ocp-schedules-15,f1dde306968a55ed,8dfd35f7f6e92227] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:46:46.606 INFO 26087 --- [ocp-schedules-15,f1dde306968a55ed,8dfd35f7f6e92227] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:46:46.611 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:46:46.611 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:46:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:46:46.623587+08:00, lastProcessTime=2025-04-08T21:46:36.623497+08:00 2025-04-08 21:46:46.656 WARN 26087 --- [ocp-async-6,c2bfc03d3f63c5fb,e2f8ed4a8b36287a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:46.656 INFO 26087 --- [ocp-async-10,26c548e5ff27fe2d,61e0884b057da45a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:46.724 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:46:46.623587+08:00 2025-04-08 21:46:46.793 INFO 26087 --- [ocp-schedules-7,768728d12a56b4a8,142c42ce94c21c07] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:46:47.649 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:47.650 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:48.271 INFO 26087 --- [prometheus-minute-schedule-1,a5ea45626dec4e26,d7045b1f798062ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:48.656 INFO 26087 --- [ocp-async-4,8ccebd09eeda82f5,a5e85ed8821205c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:48.656 WARN 26087 --- [ocp-async-2,a708e8180f5b23cf,cc8edf3750584dc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:50.210 INFO 26087 --- [ocp-schedules-10,029786673ea39eb8,915d4747e9629c9f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:46:50.211 INFO 26087 --- [ocp-schedules-10,029786673ea39eb8,915d4747e9629c9f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:46:50.650 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:50.650 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:50.656 INFO 26087 --- [ocp-async-4,1b0d1cbaab3d52e6,d4b7d4f03416fb57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:50.656 WARN 26087 --- [ocp-async-9,3b6aab627e80e215,5d439d4701f3b4aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:51.271 INFO 26087 --- [prometheus-minute-schedule-1,4072fb1d7ccfae9f,d656c9fd909a2bb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:52.656 WARN 26087 --- [ocp-async-6,24d666721f48733c,1f9fffdaae92d0aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:52.656 INFO 26087 --- [ocp-async-8,bec5d227f2c9017d,6a1796db998da6a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:53.396 INFO 26087 --- [ocp-schedules-11,3b61cc2150880743,d65223bfdfa94130] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:46:53.650 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:53.651 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:54.271 INFO 26087 --- [prometheus-minute-schedule-1,48c51063a70e96e9,70786fa289190928] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:54.656 WARN 26087 --- [ocp-async-10,9454ebbd702ceed4,58d33fd75892dad1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:54.656 INFO 26087 --- [ocp-async-2,034cadea7c1d657a,1f67f7f35827ec91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:56.442 INFO 26087 --- [ocp-schedules-1,db93aa9328701757,2a0e10ca405a0ed6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:46:56.442 INFO 26087 --- [ocp-schedules-25,c28634f835d95c1f,f3007197e8efff5a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:46:56.495 INFO 26087 --- [ocp-async-3,1696da2b3ec56d56,7ce0b51456a09f47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:46:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:46:56.623927+08:00, lastProcessTime=2025-04-08T21:46:46.623587+08:00 2025-04-08 21:46:56.651 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:56.651 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:46:56.656 WARN 26087 --- [ocp-async-9,d4afc39cc412e5ba,7aae31a9349f57b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:56.656 INFO 26087 --- [ocp-async-7,9b931a7a6ccd10f0,e01016042b97c1ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:56.714 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:46:56.623927+08:00 2025-04-08 21:46:56.794 INFO 26087 --- [ocp-schedules-9,62c9d722f8e3b04e,8b237c4666345470] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:46:57.271 INFO 26087 --- [prometheus-minute-schedule-1,a1f54a4882d27563,ab89f87142996160] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120020, interval=60 2025-04-08 21:46:58.471 INFO 26087 --- [ocp-schedules-28,ff3fff97a62b38d1,0bc3c4a65c5f5834] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:46:58.477 INFO 26087 --- [ocp-schedules-28,ff3fff97a62b38d1,0bc3c4a65c5f5834] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:46:58.477 INFO 26087 --- [ocp-schedules-28,ff3fff97a62b38d1,0bc3c4a65c5f5834] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:46:58.657 WARN 26087 --- [ocp-async-6,8880c36efd2bea65,47e8d307685fb9c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:46:58.657 INFO 26087 --- [ocp-async-6,6596170a6f800467,6ef963e4cf89d029] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:46:59.651 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:46:59.652 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:00.217 INFO 26087 --- [ocp-schedules-18,1e6d3d2feed2c36a,b68c0c614d59627a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:47:00.218 INFO 26087 --- [ocp-schedules-18,1e6d3d2feed2c36a,b68c0c614d59627a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:47:00.271 INFO 26087 --- [prometheus-minute-schedule-1,f0ae4d394592f391,2e42b91f5a657f70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:00.657 INFO 26087 --- [ocp-async-10,41f6b6d5604c9bc3,95888ff2b3c570fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:00.657 WARN 26087 --- [ocp-async-2,39cbf14155ffa462,036b1dab409318f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:02.652 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:02.653 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:02.657 WARN 26087 --- [ocp-async-4,11d1f9fbccf89cbb,3ce3848c7ca4737c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:02.657 INFO 26087 --- [ocp-async-1,4f02ce1baff5bdc6,3242a7db680fa5d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:03.271 INFO 26087 --- [prometheus-minute-schedule-1,cd6bf4b0d41132a5,b950a920811562d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:03.418 INFO 26087 --- [ocp-schedules-4,008e0ba3a50ee2d2,e59e82ac5a6f95e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:47:04.657 WARN 26087 --- [ocp-async-8,b62549a08f8993e0,1f3aa5781a1ec43a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:04.657 INFO 26087 --- [ocp-async-7,932fb66196c72616,b4ce9c32e8160556] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:05.653 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:05.653 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:06.271 INFO 26087 --- [prometheus-minute-schedule-1,088d2042acf0493a,dd9e8ae30ebacb05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:06.272 INFO 26087 --- [prometheus-cache-monitor1,4af7b75ebdc275d6,483799a755846a04] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:47:06.272 INFO 26087 --- [prometheus-cache-monitor1,4af7b75ebdc275d6,483799a755846a04] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:47:06.495 INFO 26087 --- [ocp-async-5,60a6f023364ced76,7532a1a253336d97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:47:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:47:06.622699+08:00, lastProcessTime=2025-04-08T21:46:56.623927+08:00 2025-04-08 21:47:06.657 WARN 26087 --- [ocp-async-2,2efdf9f5f15a7a3a,4006c4a8b86da1b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:06.657 INFO 26087 --- [ocp-async-6,1149990576438ee5,f5621c4c8f31543f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:06.710 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:47:06.622699+08:00 2025-04-08 21:47:06.796 INFO 26087 --- [ocp-schedules-4,76d0062cb6969aba,fd18812dd1da08c0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:47:08.653 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:08.654 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:08.657 WARN 26087 --- [ocp-async-9,9102cfc15db3c2ec,e19c98e697f89f3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:08.657 INFO 26087 --- [ocp-async-6,0bc84b813a17f3a6,d8ebe221176b045d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:09.271 INFO 26087 --- [prometheus-minute-schedule-1,02653b226b570bc8,0c1e6732e0bee5d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:10.002 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:47:10.002 INFO 26087 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:47:10.223 INFO 26087 --- [ocp-schedules-30,76235bd5f23fbaff,f6596e0a4cf0c197] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:47:10.224 INFO 26087 --- [ocp-schedules-30,76235bd5f23fbaff,f6596e0a4cf0c197] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:47:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:47:10.657 WARN 26087 --- [ocp-async-7,117c136ee7b839ef,53697761aca3803c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:10.657 INFO 26087 --- [ocp-async-6,c6f79feefc2b51df,0ccb29f4cfa65f20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:11.654 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:11.655 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:12.271 INFO 26087 --- [prometheus-minute-schedule-1,4966c52e577501f4,bf81ff5757913c6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:12.499 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744119960, nodeCount=380, affectRows=380 2025-04-08 21:47:12.501 INFO 26087 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744119960, nodeCount=377, validCount=377 2025-04-08 21:47:12.658 WARN 26087 --- [ocp-async-10,783d77ed74bee461,2b2f89d5aec2c067] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:12.658 INFO 26087 --- [ocp-async-2,f3744193eb5f2fec,baca23a4ce979d35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:13.446 INFO 26087 --- [ocp-schedules-21,ae963b0e1235fa85,c4d9a34ae25262b4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:47:14.655 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:14.655 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:14.658 WARN 26087 --- [ocp-async-9,db0bebffb1f350a3,15e04fa17c0638d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:14.658 INFO 26087 --- [ocp-async-2,9e9990f1eea44bdb,0d6965384611c41d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:15.271 INFO 26087 --- [prometheus-minute-schedule-1,ea96f16bf652cf97,d94b27e21f0fd2da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:16.433 INFO 26087 --- [ocp-schedules-27,e97033704844f8d7,df610026b9261ffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:47:16.434 INFO 26087 --- [ocp-schedules-13,46ef8c425aa30200,8ab47e793d48d833] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:47:16.437 INFO 26087 --- [ocp-schedules-28,0fb39af630ce68c3,c20ad5e29dc54d50] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:47:16.437 INFO 26087 --- [ocp-schedules-28,0fb39af630ce68c3,c20ad5e29dc54d50] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:47:16.437 INFO 26087 --- [ocp-schedules-9,bd2031705191e1ee,7e7888bfe3386292] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:47:16.438 INFO 26087 --- [ocp-schedules-27,e97033704844f8d7,df610026b9261ffc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:47:16.439 INFO 26087 --- [ocp-schedules-10,7a4b748eec701cc8,6bf4c583f9c91b39] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:47:16.439 INFO 26087 --- [ocp-schedules-10,7a4b748eec701cc8,6bf4c583f9c91b39] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:47:16.440 INFO 26087 --- [ocp-schedules-7,8f235c4ccd90e5ba,6c03fd40e90c0f3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:47:16.440 INFO 26087 --- [ocp-schedules-7,8f235c4ccd90e5ba,6c03fd40e90c0f3b] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:47:16.440 INFO 26087 --- [ocp-schedules-13,46ef8c425aa30200,8ab47e793d48d833] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:47:16.441 INFO 26087 --- [ocp-schedules-24,8616b3ef131910dd,017baca2c5d3a423] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:47:16.441 INFO 26087 --- [ocp-schedules-14,b073b81eaf1a6e3d,8d7a1debc1236763] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:47:16.441 INFO 26087 --- [ocp-schedules-14,b073b81eaf1a6e3d,8d7a1debc1236763] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:16.441 INFO 26087 --- [ocp-schedules-24,8616b3ef131910dd,017baca2c5d3a423] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:47:16.487 INFO 26087 --- [ocp-schedules-5,7af1b7de2674dc60,4e62f9a3f55640a1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:47:16.495 INFO 26087 --- [ocp-async-7,53fdecbc50c28abd,a0ef0710c1a1e24e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:47:16.506 INFO 26087 --- [ocp-schedules-4,9b5f9d3d02669b6a,a0ce49e2421d1009] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:47:16.506 INFO 26087 --- [ocp-schedules-4,9b5f9d3d02669b6a,a0ce49e2421d1009] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:47:16.545 INFO 26087 --- [ocp-schedules-21,f75bea1b6fb18a77,cd8d8b61685f7ab2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:47:16.594 INFO 26087 --- [ocp-schedules-28,653d9002ecdc149c,cf06adb99c42cc13] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:47:16.594 INFO 26087 --- [ocp-schedules-28,653d9002ecdc149c,cf06adb99c42cc13] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:47:16.594 INFO 26087 --- [ocp-schedules-29,de6adfa5574862f2,caa1a498188e49f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:47:16.601 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-04-08 21:47:16.601 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-04-08 21:47:16.605 INFO 26087 --- [ocp-schedules-10,aeeeaf1db2b83c25,558eba1a189b679c] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:47:16.605 INFO 26087 --- [ocp-schedules-10,aeeeaf1db2b83c25,558eba1a189b679c] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:47:16.606 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-04-08 21:47:16.606 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-04-08 21:47:16.606 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-04-08 21:47:16.607 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-04-08 21:47:16.607 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-04-08 21:47:16.607 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-04-08 21:47:16.609 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-04-08 21:47:16.609 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-04-08 21:47:16.609 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-04-08 21:47:16.611 INFO 26087 --- [ocp-schedules-6,6f3cacd0b307670d,8578e051100130f3] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-04-08 21:47:16.611 INFO 26087 --- [ocp-schedules-15,f1dde306968a55ed,8dfd35f7f6e92227] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:47:16.611 INFO 26087 --- [ocp-schedules-15,933ba5a946a99d18,b4ce7259876e082d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:47:16.612 INFO 26087 --- [ocp-schedules-15,933ba5a946a99d18,b4ce7259876e082d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:16.617 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:47:16.617 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:47:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:47:16.624483+08:00, lastProcessTime=2025-04-08T21:47:06.622699+08:00 2025-04-08 21:47:16.628 INFO 26087 --- [ocp-schedules-9,530b261fc8319055,1b6d8ac6f801c733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:47:16.630 INFO 26087 --- [ocp-schedules-9,530b261fc8319055,1b6d8ac6f801c733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:47:16.630 INFO 26087 --- [ocp-schedules-9,530b261fc8319055,1b6d8ac6f801c733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:47:16.648 INFO 26087 --- [ocp-schedules-22,fa6cac9ae624858f,8182894a6500f5e1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:47:16.649 INFO 26087 --- [ocp-schedules-19,42b36b5b2c2711f2,b3ed721fb2b17f2c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:47:16.649+08:00, clusters=[1000003] 2025-04-08 21:47:16.649 INFO 26087 --- [ocp-schedules-19,42b36b5b2c2711f2,b3ed721fb2b17f2c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:47:16.658 WARN 26087 --- [ocp-async-10,07bc30e6800b2327,a9d7e64cf8f0c3d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:16.658 INFO 26087 --- [ocp-async-8,0073dcd1aeaf4aa4,fe3d9b711765091b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:16.662 INFO 26087 --- [ocp-schedules-27,8648586339854e39,b8e935f54a66a6b8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:47:16.662+08:00, clusters=[1000003] 2025-04-08 21:47:16.662 INFO 26087 --- [ocp-schedules-1,14963fe5ab334edc,dc7338e8e7d72d2e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:47:16.663 INFO 26087 --- [ocp-schedules-1,14963fe5ab334edc,dc7338e8e7d72d2e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:47:16.664 INFO 26087 --- [ocp-schedules-27,8648586339854e39,b8e935f54a66a6b8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:47:16.667 INFO 26087 --- [ocp-schedules-1,14963fe5ab334edc,dc7338e8e7d72d2e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:47:16.667 INFO 26087 --- [ocp-schedules-1,14963fe5ab334edc,dc7338e8e7d72d2e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:47:16.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:47:16.624483+08:00 2025-04-08 21:47:16.797 INFO 26087 --- [ocp-schedules-8,dbf75e5a477957bf,b3936280d2e54226] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:47:16.827 INFO 26087 --- [ocp-schedules-11,9709dea297656c3b,2f98768850a22b3c] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:47:16.828 INFO 26087 --- [ocp-schedules-11,9709dea297656c3b,2f98768850a22b3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:16.841 ERROR 26087 --- [ocp-schedules-11,9709dea297656c3b,2f98768850a22b3c] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:47:16.841 INFO 26087 --- [ocp-schedules-11,9709dea297656c3b,2f98768850a22b3c] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:47:16.854 INFO 26087 --- [ocp-schedules-23,3ffee59fbe6a3a91,30163a4f8cdc130b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:47:16.854 INFO 26087 --- [ocp-schedules-23,3ffee59fbe6a3a91,30163a4f8cdc130b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:47:16.854 INFO 26087 --- [ocp-schedules-23,3ffee59fbe6a3a91,30163a4f8cdc130b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:47:16.854 INFO 26087 --- [ocp-schedules-23,3ffee59fbe6a3a91,30163a4f8cdc130b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:47:16.854 INFO 26087 --- [ocp-schedules-23,3ffee59fbe6a3a91,30163a4f8cdc130b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:47:16.854 INFO 26087 --- [ocp-schedules-23,3ffee59fbe6a3a91,30163a4f8cdc130b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:47:16.882 INFO 26087 --- [ocp-schedules-20,9d16936441a53cb4,1a9edde173261ab5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:47:17.003 INFO 26087 --- [ocp-schedules-26,90008db763c0607e,4fd4736242771316] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:47:17.004 INFO 26087 --- [ocp-schedules-26,90008db763c0607e,4fd4736242771316] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:17.019 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:47:17.020 ERROR 26087 --- [ocp-schedules-26,90008db763c0607e,4fd4736242771316] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:47:17.020 INFO 26087 --- [ocp-schedules-26,90008db763c0607e,4fd4736242771316] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:47:17.117 INFO 26087 --- [ocp-schedules-29,b799d5ad545be2da,fef20436b3082048] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:47:17.656 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:17.656 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:17.714 INFO 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:47:17.714 INFO 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:47:17.717 INFO 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:47:17.718 WARN 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:47:17.718 INFO 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:47:17.720 WARN 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:47:17.723 INFO 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:47:17.724 WARN 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:47:17.725 ERROR 26087 --- [ocp-schedules-12,aab45499174163ee,d2d927a5118f46fa] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:47:18.271 INFO 26087 --- [prometheus-minute-schedule-1,d6ad33616170b4a7,8efa1359808126ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:18.479 INFO 26087 --- [ocp-schedules-20,157e12513a4ae146,55df8be5db56096d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:47:18.492 INFO 26087 --- [ocp-schedules-20,157e12513a4ae146,55df8be5db56096d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:47:18.492 INFO 26087 --- [ocp-schedules-20,157e12513a4ae146,55df8be5db56096d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:47:18.658 WARN 26087 --- [ocp-async-8,eae4e2518569d8fc,4915464acf609fd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:18.658 INFO 26087 --- [ocp-async-9,88d8eadd0e1a622e,db77f2c66f7c6e65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:20.234 INFO 26087 --- [ocp-schedules-26,f63de198c6d43134,08ef15a5b81eca06] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:47:20.235 INFO 26087 --- [ocp-schedules-26,f63de198c6d43134,08ef15a5b81eca06] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:47:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:47:20.656 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:20.657 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:20.658 WARN 26087 --- [ocp-async-4,1aad416d9aa7635a,229d8cab3f7962b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:20.658 INFO 26087 --- [ocp-async-7,c63acc4b05548968,ef9a47616a2a503d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:21.271 INFO 26087 --- [prometheus-minute-schedule-1,e9e25c6cbcb25b75,40bdb95c22d26294] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:22.658 WARN 26087 --- [ocp-async-7,f9184f6d332ab24b,5e8854fb26ca8d50] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:22.658 INFO 26087 --- [ocp-async-10,e6b89788de35a0f5,9ac364d1f424401d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:23.466 INFO 26087 --- [ocp-schedules-7,b56b06343a85d132,5b5d056a88738232] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:47:23.657 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:23.657 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:24.271 INFO 26087 --- [prometheus-minute-schedule-1,8673e8001afa1270,36595623d474efde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:24.658 WARN 26087 --- [ocp-async-8,23d74a156527c572,f50d624af416d572] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:24.658 INFO 26087 --- [ocp-async-10,00b1788658c77b6b,dd074234f8c2a822] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:26.442 INFO 26087 --- [ocp-schedules-14,b073b81eaf1a6e3d,8d7a1debc1236763] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:47:26.442 INFO 26087 --- [ocp-schedules-24,8616b3ef131910dd,017baca2c5d3a423] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:47:26.495 INFO 26087 --- [ocp-async-10,75398a4e13b4f259,cd77fe2520e93c02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:47:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:47:26.622709+08:00, lastProcessTime=2025-04-08T21:47:16.624483+08:00 2025-04-08 21:47:26.658 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:26.658 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:26.659 WARN 26087 --- [ocp-async-9,375dc711d793cb13,57255046cf192bb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:26.659 INFO 26087 --- [ocp-async-5,03550ca962ea1406,366b7b66e486fa77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:26.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:47:26.622709+08:00 2025-04-08 21:47:26.798 INFO 26087 --- [ocp-schedules-16,9360f4ae0052aba6,57f017fce39fb8d1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:47:27.271 INFO 26087 --- [prometheus-minute-schedule-1,531cb6d83372247a,8dae1991577b1097] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:28.659 WARN 26087 --- [ocp-async-6,1e4f47f1d94c034b,704b2bfc713380e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:28.659 INFO 26087 --- [ocp-async-5,145a744aa0198279,658c6ffb81e3fa3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:29.658 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:29.659 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:30.246 INFO 26087 --- [ocp-schedules-10,1fa930727c2037da,033eb51156dd1eb0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:47:30.247 INFO 26087 --- [ocp-schedules-10,1fa930727c2037da,033eb51156dd1eb0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:47:30.271 INFO 26087 --- [prometheus-minute-schedule-1,83637130e38453f1,3403ca8e30cc8129] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:30.659 WARN 26087 --- [ocp-async-5,2fdfa46a6aa5545f,354f24e1938c0ad5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:30.659 INFO 26087 --- [ocp-async-1,74247abb355b6c55,b1d4530575d6fbc9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:32.659 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:32.659 WARN 26087 --- [ocp-async-7,0b6071ce51e4d99d,4217a43b68f9a262] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:32.659 INFO 26087 --- [ocp-async-1,815fbf948b2dcd56,ce1d382a53e64501] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:32.660 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:33.271 INFO 26087 --- [prometheus-minute-schedule-1,5d507675d15a18d3,6176e276f67e24ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:33.493 INFO 26087 --- [ocp-schedules-14,792a1b289454d281,0cb8b1f45ef42f2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:47:34.659 WARN 26087 --- [ocp-async-6,1f1c27fe2e731aa4,cd3ee93abfec628e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:34.659 INFO 26087 --- [ocp-async-3,067fab57405c4008,fa261a4926f62fff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:34.894 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:47:34.899 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:47:34.899 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=5ms, period=120000ms 2025-04-08 21:47:35.660 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:35.660 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:36.271 INFO 26087 --- [prometheus-minute-schedule-1,db1d0d8679749593,b933a978c241c333] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:36.495 INFO 26087 --- [ocp-async-8,095213762072be73,0ebaa3a8e3cd5fa1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:47:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:47:36.623946+08:00, lastProcessTime=2025-04-08T21:47:26.622709+08:00 2025-04-08 21:47:36.659 WARN 26087 --- [ocp-async-5,9902911131463a41,0f8d7706ee40c064] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:36.659 INFO 26087 --- [ocp-async-10,238bb1863ff7cb41,c920984d5eb715c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:36.707 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:47:36.623946+08:00 2025-04-08 21:47:36.799 INFO 26087 --- [ocp-schedules-16,3c87f60af1284f5e,0bdc5cdc8e0c668b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:47:38.493 INFO 26087 --- [ocp-schedules-10,82dee2f9b19c286f,c17a7621d3bc1057] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:47:38.542 INFO 26087 --- [ocp-schedules-10,82dee2f9b19c286f,c17a7621d3bc1057] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:47:38.542 INFO 26087 --- [ocp-schedules-10,82dee2f9b19c286f,c17a7621d3bc1057] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:47:38.660 WARN 26087 --- [ocp-async-7,2b0ab79f65f803cf,dc13408a88208d79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:38.660 INFO 26087 --- [ocp-async-1,c47a389bf6654263,1fd6a217044f0855] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:38.660 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:38.661 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:39.271 INFO 26087 --- [prometheus-minute-schedule-1,988c5ccd13a0c164,34c9440a8ef2a4c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:40.001 INFO 26087 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:47:40.001 INFO 26087 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:47:40.254 INFO 26087 --- [ocp-schedules-18,a31eef5da0489693,99601b3031b6170c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:47:40.255 INFO 26087 --- [ocp-schedules-18,a31eef5da0489693,99601b3031b6170c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:47:40.660 WARN 26087 --- [ocp-async-1,7127f0bfd49d5090,85fc67b2de47cca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:40.660 INFO 26087 --- [ocp-async-3,f79b6046506458a1,ea979274c25cc48e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:41.661 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:41.662 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:42.271 INFO 26087 --- [prometheus-minute-schedule-1,8034918a99c6c89d,0b6dfb9f0252a4a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:42.660 WARN 26087 --- [ocp-async-4,772491ead7acc072,a6c8de6f22f77a93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:42.660 INFO 26087 --- [ocp-async-5,b476cf3389d17474,7aaae298a77783fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:43.514 INFO 26087 --- [ocp-schedules-16,3d30abd8b65ec0ef,0e15c1b2e23bdc6e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:47:44.660 INFO 26087 --- [ocp-async-5,3ff9985151395baf,231d7e442961e5c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:44.660 WARN 26087 --- [ocp-async-9,7f49b38cfcd3bbb1,bd530db11a64d400] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:44.662 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:44.662 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:45.271 INFO 26087 --- [prometheus-minute-schedule-1,a97009badffabe80,4a85466ee453c43a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:46.437 INFO 26087 --- [ocp-schedules-28,b592cd63cfbbc150,79704ed36636908d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:47:46.438 INFO 26087 --- [ocp-schedules-6,a1033802e7a55aa1,a7268b968670cc36] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:47:46.438 INFO 26087 --- [ocp-schedules-6,a1033802e7a55aa1,a7268b968670cc36] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:47:46.439 INFO 26087 --- [ocp-schedules-26,7361fe1d70159320,1f1254681156c661] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:47:46.439 INFO 26087 --- [ocp-schedules-26,7361fe1d70159320,1f1254681156c661] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:47:46.440 INFO 26087 --- [ocp-schedules-29,53ba0d8db11d1c37,3161ae9ca6752497] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:47:46.440 INFO 26087 --- [ocp-schedules-29,53ba0d8db11d1c37,3161ae9ca6752497] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:47:46.440 INFO 26087 --- [ocp-schedules-3,bec18e586737e269,9bb6ed826026e48b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:47:46.441 INFO 26087 --- [ocp-schedules-11,feb4b8fb14e4cce2,826066513d092f60] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:47:46.441 INFO 26087 --- [ocp-schedules-30,8349b9c39f5d1e7e,e4f1f53f460b77d1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:47:46.441 INFO 26087 --- [ocp-schedules-11,feb4b8fb14e4cce2,826066513d092f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:46.441 INFO 26087 --- [ocp-schedules-30,8349b9c39f5d1e7e,e4f1f53f460b77d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:47:46.495 INFO 26087 --- [ocp-async-7,d53be26acc3c8afc,92a11d9e43451fb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:47:46.512 INFO 26087 --- [ocp-schedules-27,4b98db6b0858496b,5c54f95b5e6c19a4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:47:46.513 INFO 26087 --- [ocp-schedules-27,4b98db6b0858496b,5c54f95b5e6c19a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:46.513 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:47:46.596 INFO 26087 --- [ocp-schedules-10,da7ba7c6c3131c54,68226e55801b3bf9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:47:46.617 INFO 26087 --- [ocp-schedules-15,933ba5a946a99d18,b4ce7259876e082d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:47:46.617 INFO 26087 --- [ocp-schedules-15,6292f3d89e653c83,7520b886dfdd4f0b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:47:46.618 INFO 26087 --- [ocp-schedules-15,6292f3d89e653c83,7520b886dfdd4f0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:47:46.623 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:47:46.623 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:47:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:47:46.623299+08:00, lastProcessTime=2025-04-08T21:47:36.623946+08:00 2025-04-08 21:47:46.660 WARN 26087 --- [ocp-async-1,4e52a2f093d5a3d3,6833fff217373809] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:46.660 INFO 26087 --- [ocp-async-1,6c5b87bd55f67497,10a250f18706feab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:46.720 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:47:46.623299+08:00 2025-04-08 21:47:46.801 INFO 26087 --- [ocp-schedules-25,4c509afa0d91091d,c795171abf343bad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:47:47.662 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:47.663 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:48.271 INFO 26087 --- [prometheus-minute-schedule-1,ab66a42e87db9385,07d2d9bea313770f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:48.660 WARN 26087 --- [ocp-async-1,e89b1934d41b5f08,cc898a1eae710c0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:48.660 INFO 26087 --- [ocp-async-10,5e12bb833397674d,484e809f811e7fd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:50.278 INFO 26087 --- [ocp-schedules-22,e62123d0470dcb44,e00d2970cc75b81d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:47:50.280 INFO 26087 --- [ocp-schedules-22,e62123d0470dcb44,e00d2970cc75b81d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:47:50.660 WARN 26087 --- [ocp-async-5,9777909b02d3e503,c9ea415a42c2dfd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:50.661 INFO 26087 --- [ocp-async-7,20cb21f2ebe4bdaf,9e3ecaa5077e9ba3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:50.663 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:50.663 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:51.271 INFO 26087 --- [prometheus-minute-schedule-1,d0473eddc9de8f3e,9cbef549f5349b68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:52.661 WARN 26087 --- [ocp-async-8,0bb1c439fd587969,cedcdba6a45504a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:52.661 INFO 26087 --- [ocp-async-7,2366eb6604a22dde,c6850b01f7cfeae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:53.543 INFO 26087 --- [ocp-schedules-16,a4d3b4d59bdbfc67,3e1d75cec6da3222] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:47:53.663 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:53.664 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:54.271 INFO 26087 --- [prometheus-minute-schedule-1,9ba70af30fcc5acb,6f0f7526b01f62ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:54.661 WARN 26087 --- [ocp-async-3,cae399f8a29f8b8f,bca5647719521a84] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:54.661 INFO 26087 --- [ocp-async-4,7fa11e831ed3fb16,b8d9f1dabba702ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:56.442 INFO 26087 --- [ocp-schedules-30,8349b9c39f5d1e7e,e4f1f53f460b77d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:47:56.442 INFO 26087 --- [ocp-schedules-11,feb4b8fb14e4cce2,826066513d092f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:47:56.496 INFO 26087 --- [ocp-async-1,4b062edef21cc2d2,5bced95158a637cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:47:56.513 INFO 26087 --- [ocp-schedules-27,4b98db6b0858496b,5c54f95b5e6c19a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:47:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:47:56.623227+08:00, lastProcessTime=2025-04-08T21:47:46.623299+08:00 2025-04-08 21:47:56.661 WARN 26087 --- [ocp-async-6,cb80ad6674fd6ade,9eef781ec293ab6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:56.661 INFO 26087 --- [ocp-async-10,b82c43381d59a204,ff2a8145037ad1d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:56.664 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:56.664 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:47:56.723 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:47:56.623227+08:00 2025-04-08 21:47:56.802 INFO 26087 --- [ocp-schedules-19,378a7588c5c6f529,7216961a4d6f4c11] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:47:57.271 INFO 26087 --- [prometheus-minute-schedule-1,d4e6d096cdf41a92,208bd2edd7332c44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120080, interval=60 2025-04-08 21:47:58.544 INFO 26087 --- [ocp-schedules-13,e2aa3a1fec9cb712,e10c5811ba8334ac] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:47:58.549 INFO 26087 --- [ocp-schedules-13,e2aa3a1fec9cb712,e10c5811ba8334ac] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:47:58.549 INFO 26087 --- [ocp-schedules-13,e2aa3a1fec9cb712,e10c5811ba8334ac] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:47:58.661 INFO 26087 --- [ocp-async-8,1b0486741e410010,52324690c57f7a9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:47:58.661 WARN 26087 --- [ocp-async-2,71802d32498047cf,61567a94ac1ecd70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:47:59.664 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:47:59.665 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:00.271 INFO 26087 --- [prometheus-minute-schedule-1,b3a30a42ff1bb713,30de736e5ce9af88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:00.325 INFO 26087 --- [ocp-schedules-26,aaf000e9f800d122,37f4bec8a1891c8b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:48:00.327 INFO 26087 --- [ocp-schedules-26,aaf000e9f800d122,37f4bec8a1891c8b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:48:00.661 WARN 26087 --- [ocp-async-1,23b3348e08728795,73f5dc40b2b21a1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:00.661 INFO 26087 --- [ocp-async-3,42aaedd01580993b,40c6d2cfb3467810] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:02.661 WARN 26087 --- [ocp-async-5,32756ada7ed35e84,ae5117bde3b9ef43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:02.661 INFO 26087 --- [ocp-async-6,5559688784db1d19,1ee2613dee5992e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:02.665 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:02.666 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:03.271 INFO 26087 --- [prometheus-minute-schedule-1,005ea2c697b4d316,b27dd0da48fa55c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:03.564 INFO 26087 --- [ocp-schedules-21,becfe7c7da7a97ef,3e4cfe0f45a01d9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:48:04.661 WARN 26087 --- [ocp-async-2,fa173ce4417fb07c,37509a887a77825f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:04.661 INFO 26087 --- [ocp-async-10,5fad4f9d041f54a7,d596733fe8c17d41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:05.666 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:05.666 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:06.271 INFO 26087 --- [prometheus-minute-schedule-1,71d82d1c3d933b67,3b9d78a51ef12e23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:06.272 INFO 26087 --- [prometheus-cache-monitor1,413029cc53ef7d36,a6df5578108e4933] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:48:06.272 INFO 26087 --- [prometheus-cache-monitor1,413029cc53ef7d36,a6df5578108e4933] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:48:06.496 INFO 26087 --- [ocp-async-7,43ca09e12aae6e9f,c1b51a9c1cab7827] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:48:06.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:48:06.622588+08:00, lastProcessTime=2025-04-08T21:47:56.623227+08:00 2025-04-08 21:48:06.662 WARN 26087 --- [ocp-async-9,add335dddbdaa6b4,f90913b4d72f8ef6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:06.662 INFO 26087 --- [ocp-async-1,1673220ce9295251,4b149f8def3009ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:06.705 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:48:06.622588+08:00 2025-04-08 21:48:06.803 INFO 26087 --- [ocp-schedules-11,08df5ae634c5869f,b6a0506a5eda0eb7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:48:06.920 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-04-08T21:44:06.88, end=2025-04-08T21:46:06.918} 2025-04-08 21:48:06.922 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [] 2025-04-08 21:48:06.922 INFO 26087 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-04-08 21:48:08.662 WARN 26087 --- [ocp-async-5,7bc46ca680ab9087,e2af4bcba3f6ab1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:08.662 INFO 26087 --- [ocp-async-1,a6f35251ed81c7e4,90f472c78c2ae4d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:08.666 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:08.667 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:09.271 INFO 26087 --- [prometheus-minute-schedule-1,ef3aaba1ae7027ee,7835878167cae4f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:10.001 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:48:10.001 INFO 26087 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:48:10.329 INFO 26087 --- [ocp-schedules-24,ec18a9961b45d404,553812ecb2a17cdd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:48:10.330 INFO 26087 --- [ocp-schedules-24,ec18a9961b45d404,553812ecb2a17cdd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:48:10.518 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:48:10.662 WARN 26087 --- [ocp-async-2,49d092b09fd11e25,38539142bfb68039] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:10.662 INFO 26087 --- [ocp-async-1,5ebf77cd99838321,c799fa4810e1aa66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:11.667 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:11.668 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:12.271 INFO 26087 --- [prometheus-minute-schedule-1,2321bff814b8943e,a799bb935b638ea8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:12.522 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744120020, nodeCount=183, affectRows=183 2025-04-08 21:48:12.523 INFO 26087 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744120020, nodeCount=182, validCount=182 2025-04-08 21:48:12.530 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744120020, nodeCount=197, affectRows=197 2025-04-08 21:48:12.532 INFO 26087 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744120020, nodeCount=195, validCount=195 2025-04-08 21:48:12.662 WARN 26087 --- [ocp-async-1,850825afc5db19de,864c30c8d98f30af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:12.662 INFO 26087 --- [ocp-async-9,427f0e513123108a,320daa30a9e31903] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:13.604 INFO 26087 --- [ocp-schedules-20,a2b4b072c80593ee,1095d9ee2afd0db3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:48:14.662 WARN 26087 --- [ocp-async-5,bc3de151d5975b0a,f0672db0c79f2961] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:14.662 INFO 26087 --- [ocp-async-6,1a2a80585633a33d,cfe18f57b6f0c3db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:14.668 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:14.668 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:15.271 INFO 26087 --- [prometheus-minute-schedule-1,167ffa598c4f4cf1,1b89d63606f15178] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:16.433 INFO 26087 --- [ocp-schedules-2,a239076eacbb46de,d53af4e53b7640bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:48:16.434 INFO 26087 --- [ocp-schedules-8,4615ff172e606981,a05373d944621754] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:48:16.437 INFO 26087 --- [ocp-schedules-24,09df06c718995176,0eed3eb108b8fb50] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:48:16.438 INFO 26087 --- [ocp-schedules-20,f88532b8caa9e8ec,698caf83b9ba1d2e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:48:16.438 INFO 26087 --- [ocp-schedules-20,f88532b8caa9e8ec,698caf83b9ba1d2e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:48:16.440 INFO 26087 --- [ocp-schedules-4,8e2badf120c4629d,45b3926cadf34780] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:48:16.440 INFO 26087 --- [ocp-schedules-16,565236702b30679f,2b31f762de70130a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:48:16.440 INFO 26087 --- [ocp-schedules-4,8e2badf120c4629d,45b3926cadf34780] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:48:16.440 INFO 26087 --- [ocp-schedules-16,565236702b30679f,2b31f762de70130a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:48:16.440 INFO 26087 --- [ocp-schedules-2,a239076eacbb46de,d53af4e53b7640bd] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:48:16.441 INFO 26087 --- [ocp-schedules-8,4615ff172e606981,a05373d944621754] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:48:16.441 INFO 26087 --- [ocp-schedules-14,b3fd6dd1093c51d6,2e82b259ce476580] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:48:16.441 INFO 26087 --- [ocp-schedules-8,626fae9a13674ace,becd069c7a63f1a9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:48:16.441 INFO 26087 --- [ocp-schedules-8,626fae9a13674ace,becd069c7a63f1a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:16.441 INFO 26087 --- [ocp-schedules-14,b3fd6dd1093c51d6,2e82b259ce476580] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:48:16.447 INFO 26087 --- [ocp-schedules-19,dd8cd76331b4da20,f5ddb9479743a150] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-04-08 21:48:16.448 INFO 26087 --- [ocp-schedules-19,dd8cd76331b4da20,f5ddb9479743a150] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:16.453 INFO 26087 --- [ocp-schedules-27,b8019d3300656843,e94d27d9d91f9fe2] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-04-08 21:48:16.453 INFO 26087 --- [ocp-schedules-27,b8019d3300656843,e94d27d9d91f9fe2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-04-08 21:48:16.454 INFO 26087 --- [ocp-schedules-30,e8c3949263dca364,8dbd7c6eb3e856aa] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:48:16.471 INFO 26087 --- [pool-overview-container-8,e32e5968103fbea4,d6852d0fbce69011] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:48:16.471 INFO 26087 --- [pool-overview-container-1,32d2ad1084b9fad9,4765829e431b58eb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:48:16.476 INFO 26087 --- [pool-11-thread-8,32d2ad1084b9fad9,0c07fd61ae290911] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-04-08 21:48:16.476 INFO 26087 --- [pool-11-thread-6,32d2ad1084b9fad9,1349d2475c544a9b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-04-08 21:48:16.476 INFO 26087 --- [pool-11-thread-6,32d2ad1084b9fad9,1349d2475c544a9b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-04-08 21:48:16.476 INFO 26087 --- [pool-11-thread-5,32d2ad1084b9fad9,e010601327565a1f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-04-08 21:48:16.476 INFO 26087 --- [pool-11-thread-7,32d2ad1084b9fad9,39d3e197d4ea9bda] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-04-08 21:48:16.478 INFO 26087 --- [ForkJoinPool-7-worker-1,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:48:16.478 INFO 26087 --- [ForkJoinPool-7-worker-10,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-04-08 21:48:16.479 INFO 26087 --- [pool-11-thread-7,32d2ad1084b9fad9,39d3e197d4ea9bda] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-04-08 21:48:16.479 INFO 26087 --- [pool-11-thread-7,32d2ad1084b9fad9,39d3e197d4ea9bda] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-04-08 21:48:16.479 INFO 26087 --- [ForkJoinPool-7-worker-1,32d2ad1084b9fad9,1ecf5d08fa5af4e1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of ocpdb 2025-04-08 21:48:16.479 INFO 26087 --- [ForkJoinPool-7-worker-1,32d2ad1084b9fad9,1ecf5d08fa5af4e1] c.o.o.p.o.i.OasSessionStatServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000003, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getTenantSessionStat 2025-04-08 21:48:16.479 INFO 26087 --- [ForkJoinPool-7-worker-1,32d2ad1084b9fad9,1ecf5d08fa5af4e1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of ocpdb 2025-04-08 21:48:16.479 INFO 26087 --- [ForkJoinPool-7-worker-10,32d2ad1084b9fad9,516c2c68b36d10a5] c.o.o.p.s.d.internal.OutlineServiceImpl : Status:TAKINGOVER of Cluster ObClusterEntity{id=1000003, name=ocpdb, obClusterId=1744114127, status=TAKINGOVER} is abnormal, skip getOutlineHistory 2025-04-08 21:48:16.479 INFO 26087 --- [pool-11-thread-8,32d2ad1084b9fad9,0c07fd61ae290911] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-04-08 21:48:16.479 INFO 26087 --- [pool-11-thread-5,32d2ad1084b9fad9,e010601327565a1f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-04-08 21:48:16.479 INFO 26087 --- [pool-11-thread-6,32d2ad1084b9fad9,1349d2475c544a9b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-04-08 21:48:16.480 INFO 26087 --- [pool-11-thread-6,32d2ad1084b9fad9,1349d2475c544a9b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-04-08 21:48:16.496 INFO 26087 --- [ocp-async-2,2ff88b5f918a006a,1f609e2cd4b361dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:48:16.546 INFO 26087 --- [ocp-schedules-10,421750561c3df661,c45ec275b9c25ee3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:48:16.596 INFO 26087 --- [ocp-schedules-7,8a4a120b568c4915,9b3bfad5d46a942a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:48:16.596 INFO 26087 --- [ocp-schedules-7,8a4a120b568c4915,9b3bfad5d46a942a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:48:16.598 INFO 26087 --- [ocp-schedules-24,642b38ed26f05426,4d9fe68a5121598e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:48:16.606 INFO 26087 --- [ocp-schedules-4,d7c18e276eaecfd1,a7f1edb02c2ac03f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:48:16.606 INFO 26087 --- [ocp-schedules-4,d7c18e276eaecfd1,a7f1edb02c2ac03f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:48:16.623 INFO 26087 --- [ocp-schedules-15,6292f3d89e653c83,7520b886dfdd4f0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:48:16.624 INFO 26087 --- [ocp-schedules-15,2d83ad37181560c7,d5ffb79312cbb97d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:48:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:48:16.623104+08:00, lastProcessTime=2025-04-08T21:48:06.622588+08:00 2025-04-08 21:48:16.624 INFO 26087 --- [ocp-schedules-15,2d83ad37181560c7,d5ffb79312cbb97d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:16.629 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:48:16.629 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:48:16.630 INFO 26087 --- [ocp-schedules-16,76690cb634485a90,f8f69d5d02b2f4ce] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:48:16.632 INFO 26087 --- [ocp-schedules-16,76690cb634485a90,f8f69d5d02b2f4ce] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:48:16.632 INFO 26087 --- [ocp-schedules-16,76690cb634485a90,f8f69d5d02b2f4ce] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:48:16.653 INFO 26087 --- [ocp-schedules-27,d164eb570589fed8,846e6050433dce89] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:48:16.653 INFO 26087 --- [ocp-schedules-30,1637c48443e8f183,2c93e3e62628841c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:48:16.653+08:00, clusters=[1000003] 2025-04-08 21:48:16.654 INFO 26087 --- [ocp-schedules-30,1637c48443e8f183,2c93e3e62628841c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:48:16.662 WARN 26087 --- [ocp-async-10,fa559a1c1cf3b5f8,aeef489ca3ee9547] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:16.662 INFO 26087 --- [ocp-async-4,4c7dfd027a72616a,fef03a730bcbe38e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:16.667 INFO 26087 --- [ocp-schedules-3,dbfda23122df3913,d2d9cbc4d4407817] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:48:16.667 INFO 26087 --- [ocp-schedules-3,dbfda23122df3913,d2d9cbc4d4407817] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:48:16.668 INFO 26087 --- [ocp-schedules-23,89fa604ee6eedd18,1d53964ead81d65d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:48:16.668+08:00, clusters=[1000003] 2025-04-08 21:48:16.670 INFO 26087 --- [ocp-schedules-23,89fa604ee6eedd18,1d53964ead81d65d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:48:16.672 INFO 26087 --- [ocp-schedules-3,dbfda23122df3913,d2d9cbc4d4407817] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:48:16.672 INFO 26087 --- [ocp-schedules-3,dbfda23122df3913,d2d9cbc4d4407817] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:48:16.733 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:48:16.623104+08:00 2025-04-08 21:48:16.767 INFO 26087 --- [ocp-schedules-17,7c3ce397ec4ee226,4e7082e16c59b112] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-04-08 21:48:16.767 INFO 26087 --- [ocp-schedules-17,7c3ce397ec4ee226,4e7082e16c59b112] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-04-08 21:48:16.773 INFO 26087 --- [pool-31-thread-1,7c3ce397ec4ee226,3ab6e879ca4171e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-04-08 21:48:16.778 WARN 26087 --- [ocp-schedules-17,7c3ce397ec4ee226,4e7082e16c59b112] t.b.t.CollectAllTenantMiniCompactionTask : Failed to collect mini compaction concurrently, errMsg:com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor, cause:{} com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createCompactionOperator(AbstractObOperatorFactory.java:316) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl.collectMiniCompaction(TenantCompactionCollectServiceImpl.java:135) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$FastClassBySpringCGLIB$$7ded023a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.obops.internal.tenant.TenantCompactionCollectServiceImpl$$EnhancerBySpringCGLIB$$14662e73.collectMiniCompaction() at com.oceanbase.ocp.service.task.business.tenant.CollectAllTenantMiniCompactionTask.lambda$run$1(CollectAllTenantMiniCompactionTask.java:46) at com.oceanbase.ocp.core.thread.ConcurrentExecutor.lambda$exec$2(ConcurrentExecutor.java:73) at com.oceanbase.ocp.core.thread.LocaleDecorator.lambda$decorate$1(LocaleDecorator.java:41) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$1(TraceDecorator.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-04-08 21:48:16.778 INFO 26087 --- [ocp-schedules-17,7c3ce397ec4ee226,4e7082e16c59b112] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-04-08 21:48:16.805 INFO 26087 --- [ocp-schedules-25,0c3df34ef5de205a,e45fedcfe0732349] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:48:16.855 INFO 26087 --- [ocp-schedules-5,a283bbecad21ef5b,9dd13040af8431c2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:48:16.855 INFO 26087 --- [ocp-schedules-5,a283bbecad21ef5b,9dd13040af8431c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:48:16.855 INFO 26087 --- [ocp-schedules-5,a283bbecad21ef5b,9dd13040af8431c2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:48:16.855 INFO 26087 --- [ocp-schedules-5,a283bbecad21ef5b,9dd13040af8431c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:48:16.855 INFO 26087 --- [ocp-schedules-5,a283bbecad21ef5b,9dd13040af8431c2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:48:16.855 INFO 26087 --- [ocp-schedules-5,a283bbecad21ef5b,9dd13040af8431c2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:48:16.882 INFO 26087 --- [ocp-schedules-29,71ed93bb8ee8128f,cc76db5b253f801d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-04-08 21:48:16.883 INFO 26087 --- [ocp-schedules-29,71ed93bb8ee8128f,cc76db5b253f801d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:16.893 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:48:16.893 INFO 26087 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-04-08 21:48:17.020 INFO 26087 --- [ocp-schedules-10,89a3adb5253c7ad0,cf206750367695dc] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:48:17.021 INFO 26087 --- [ocp-schedules-10,89a3adb5253c7ad0,cf206750367695dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:17.036 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:48:17.038 ERROR 26087 --- [ocp-schedules-10,89a3adb5253c7ad0,cf206750367695dc] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:48:17.038 INFO 26087 --- [ocp-schedules-10,89a3adb5253c7ad0,cf206750367695dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:48:17.103 INFO 26087 --- [ocp-schedules-1,c89421910bcb3e4a,c2d6cb631fabde60] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-04-08 21:48:17.109 INFO 26087 --- [ocp-schedules-1,c89421910bcb3e4a,c2d6cb631fabde60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:48:17.110 WARN 26087 --- [ocp-schedules-1,c89421910bcb3e4a,c2d6cb631fabde60] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:48:17.110 INFO 26087 --- [ocp-schedules-1,c89421910bcb3e4a,c2d6cb631fabde60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:48:17.111 WARN 26087 --- [ocp-schedules-1,c89421910bcb3e4a,c2d6cb631fabde60] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:48:17.114 INFO 26087 --- [ocp-schedules-1,c89421910bcb3e4a,c2d6cb631fabde60] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=1, cachedSize=7 2025-04-08 21:48:17.124 INFO 26087 --- [ocp-schedules-20,ba7fd6a9ea8aed76,94ef2844460eba8d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:48:17.668 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:17.669 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:17.728 INFO 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:48:17.728 INFO 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:48:17.730 INFO 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:48:17.731 WARN 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:48:17.731 INFO 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:48:17.732 WARN 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:48:17.735 INFO 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:48:17.737 WARN 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:48:17.738 ERROR 26087 --- [ocp-schedules-2,fcfe335b304503e2,b605c32fdebe130e] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:48:18.271 INFO 26087 --- [prometheus-minute-schedule-1,b07d01e171e4b85e,acc0ee3c4bda07f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:18.550 INFO 26087 --- [ocp-schedules-13,627ea79ddf821f6d,7a23741f2f0144d8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:48:18.561 INFO 26087 --- [ocp-schedules-13,627ea79ddf821f6d,7a23741f2f0144d8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:48:18.561 INFO 26087 --- [ocp-schedules-13,627ea79ddf821f6d,7a23741f2f0144d8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:48:18.662 WARN 26087 --- [ocp-async-9,05234ef48db457b2,250c28a7a056aebe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:18.662 INFO 26087 --- [ocp-async-5,f656cdc41d6ce6c6,d45562e15f631a70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:20.334 INFO 26087 --- [ocp-schedules-3,df124eb8f48fc2b0,6ace9eeefb5e3d34] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:48:20.335 INFO 26087 --- [ocp-schedules-3,df124eb8f48fc2b0,6ace9eeefb5e3d34] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:48:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:48:20.663 WARN 26087 --- [ocp-async-2,a7360d0c897dc376,8c8ec479d9c1b8f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:20.663 INFO 26087 --- [ocp-async-5,7649d4678493c797,ebbe42683678302d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:20.669 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:20.670 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:21.271 INFO 26087 --- [prometheus-minute-schedule-1,ecd66c4981d36406,a9e6e3aa6406a139] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:22.663 INFO 26087 --- [ocp-async-10,2581c1f3e69e33a5,a28e672bd8a66aa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:22.663 WARN 26087 --- [ocp-async-8,d0cd4e6505578662,f98e8e87b93a9996] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:23.623 INFO 26087 --- [ocp-schedules-23,c221bf710a063f71,bcdbdac860ccecc8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:48:23.670 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:23.670 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:24.271 INFO 26087 --- [prometheus-minute-schedule-1,b79a8f6271b4507b,c25230aab7483f38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:24.663 WARN 26087 --- [ocp-async-3,7026f30c0fbf66ce,151a2a5d3a5460bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:24.663 INFO 26087 --- [ocp-async-10,b59585d18c9687bd,1daa3e7da79fae89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:26.442 INFO 26087 --- [ocp-schedules-8,626fae9a13674ace,becd069c7a63f1a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:48:26.442 INFO 26087 --- [ocp-schedules-14,b3fd6dd1093c51d6,2e82b259ce476580] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:48:26.448 INFO 26087 --- [ocp-schedules-19,dd8cd76331b4da20,f5ddb9479743a150] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=1, successCount=1 2025-04-08 21:48:26.496 INFO 26087 --- [ocp-async-10,2d6539d2680bddf3,0981411a7118e1b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:48:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:48:26.622895+08:00, lastProcessTime=2025-04-08T21:48:16.623104+08:00 2025-04-08 21:48:26.663 WARN 26087 --- [ocp-async-7,c6d529778dc8e638,08b817fd28859272] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:26.663 INFO 26087 --- [ocp-async-6,5ed09ccf567cf8aa,1160cc3bebcf466e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:26.670 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:26.671 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:26.718 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:48:26.622895+08:00 2025-04-08 21:48:26.806 INFO 26087 --- [ocp-schedules-19,448a0eb89f477a3b,bb0108850e7628c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:48:27.271 INFO 26087 --- [prometheus-minute-schedule-1,10c9e027ff2e45b9,22f9af50720f7e94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:28.663 WARN 26087 --- [ocp-async-6,cc6f556fb5c28d16,f0659f2c2d042fe3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:28.663 INFO 26087 --- [ocp-async-1,f7a456b228d92362,fdc667b9a53bfbec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:29.671 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:29.671 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:30.271 INFO 26087 --- [prometheus-minute-schedule-1,be5735ac788a1ae1,cdc68cac26989cba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:30.367 INFO 26087 --- [ocp-schedules-28,9380a2e76438aa4c,7523d31d58e4f5f5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:48:30.369 INFO 26087 --- [ocp-schedules-28,9380a2e76438aa4c,7523d31d58e4f5f5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:48:30.663 WARN 26087 --- [ocp-async-4,70625a0d1c0e2238,c0fe3b5c38061989] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:30.663 INFO 26087 --- [ocp-async-10,923f1944f4424409,a3b3191653c2413b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:32.664 WARN 26087 --- [ocp-async-7,78d2c2bfb0a580e1,c5e959c4ef9d4c99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:32.664 INFO 26087 --- [ocp-async-10,05abba15d5980bf4,c1f402a934a0505c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:32.672 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:32.672 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:33.271 INFO 26087 --- [prometheus-minute-schedule-1,b5f7b5a4a8fe24ff,fe86608f3114f425] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:33.648 INFO 26087 --- [ocp-schedules-5,a5de9e3b8eb17484,979e4d89bde88f38] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:48:34.383 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-04-08 21:48:34.383 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-04-08 21:48:34.386 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:48:34.386 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:48:34.386 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:48:34.386 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:48:34.389 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-04-08 21:48:34.389 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : init spring 2025-04-08 21:48:34.389 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-04-08 21:48:34.389 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : starting 2025-04-08 21:48:34.410 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:48:34.415 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:48:34.415 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:48:34.415 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:48:34.415 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:48:34.415 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:48:34.420 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:48:34.664 WARN 26087 --- [ocp-async-5,4895a1240add23f1,e409d20ab2c6d312] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:34.664 INFO 26087 --- [ocp-async-3,bc5ba978788d8895,52beb34c2b486b89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:34.711 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-04-08 21:48:35.038 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.649230213S 2025-04-08 21:48:35.038 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-04-08 21:48:35.093 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-04-08 21:48:35.093 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-04-08 21:48:35.117 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-04-08 21:48:35.154 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-04-08 21:48:35.154 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-04-08 21:48:35.154 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-04-08 21:48:35.154 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.5 2025-04-08 21:48:35.154 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.5 2025-04-08 21:48:35.154 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-04-08 21:48:35.161 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-04-08 21:48:35.267 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] o.s.boot.SpringApplication : Started application in 0.88 seconds (JVM running for 2820.723) 2025-04-08 21:48:35.267 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.881266638S 2025-04-08 21:48:35.551 INFO 26087 --- [ocp-schedules-23,9ca7cea1d33bcda4,86a4b6f100e55bbe] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-04-08 21:48:35.672 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:35.673 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:36.271 INFO 26087 --- [prometheus-minute-schedule-1,842e45fdbbd590c2,5908a67612d6dee5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:36.496 INFO 26087 --- [ocp-async-6,216bf9e81aa8b6c6,e61eff5fe59ca705] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:48:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:48:36.623798+08:00, lastProcessTime=2025-04-08T21:48:26.622895+08:00 2025-04-08 21:48:36.664 INFO 26087 --- [ocp-async-1,5d07073fe858c344,95bb5dbc3561111a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:36.664 WARN 26087 --- [ocp-async-2,0b470e85d868eec4,e7fba04ca9a2fba0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:36.738 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:48:36.623798+08:00 2025-04-08 21:48:36.808 INFO 26087 --- [ocp-schedules-12,b83fd4ee51d5d994,4d7adb11cd8883bd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:48:38.563 INFO 26087 --- [ocp-schedules-22,f7acb5adcf49804b,a5c866f8a20f8822] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:48:38.566 INFO 26087 --- [ocp-schedules-22,f7acb5adcf49804b,a5c866f8a20f8822] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:48:38.566 INFO 26087 --- [ocp-schedules-22,f7acb5adcf49804b,a5c866f8a20f8822] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:48:38.664 WARN 26087 --- [ocp-async-8,9cd0d41ef758d7d9,c96a8e6d31f7baf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:38.664 INFO 26087 --- [ocp-async-10,0ec3e9f008db31ee,8620b10d879971fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:38.673 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:38.673 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:39.271 INFO 26087 --- [prometheus-minute-schedule-1,5fd9622efe1aa09e,006cece59c28efbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:40.001 INFO 26087 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:48:40.002 INFO 26087 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:48:40.384 INFO 26087 --- [ocp-schedules-11,0d4cfb64290b5881,73f73ff2ed8d8d99] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:48:40.385 INFO 26087 --- [ocp-schedules-11,0d4cfb64290b5881,73f73ff2ed8d8d99] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:48:40.664 WARN 26087 --- [ocp-async-10,d062ec3f2d14e32a,7b9b5864c93aea9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:40.664 INFO 26087 --- [ocp-async-3,6451c75735359953,79fabd8111d1f2c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:41.673 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:41.674 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:42.271 INFO 26087 --- [prometheus-minute-schedule-1,db42caba3ec721d8,228e976493308ae9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:42.664 WARN 26087 --- [ocp-async-2,c99daa0da895af0d,099e55d6cff3f85c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:42.664 INFO 26087 --- [ocp-async-3,93f6f6cf9d9ce4bf,a1225448ac92eba9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:43.671 INFO 26087 --- [ocp-schedules-16,1b137388b99099e6,d648a39f848e26da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:48:44.664 INFO 26087 --- [ocp-async-1,3b07b5fc239d1833,45b8e7c53f13c391] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:44.664 WARN 26087 --- [ocp-async-8,f3c130084d80bd06,9cfa7824f8c4d202] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:44.674 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:44.674 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:45.271 INFO 26087 --- [prometheus-minute-schedule-1,77e1756829ce8302,b2630ea2db499c88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:46.437 INFO 26087 --- [ocp-schedules-2,dd4a48bd077c9b8b,930b11324d38bae2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:48:46.438 INFO 26087 --- [ocp-schedules-23,9898a51d4469122f,d08c5c9d6b9c6f45] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:48:46.438 INFO 26087 --- [ocp-schedules-23,9898a51d4469122f,d08c5c9d6b9c6f45] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:48:46.440 INFO 26087 --- [ocp-schedules-7,8b752882af7dda17,0f95c100393557b8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:48:46.440 INFO 26087 --- [ocp-schedules-7,8b752882af7dda17,0f95c100393557b8] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:48:46.440 INFO 26087 --- [ocp-schedules-5,ea4943c933f8243a,e2ec0c3056ffbf7a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:48:46.440 INFO 26087 --- [ocp-schedules-5,ea4943c933f8243a,e2ec0c3056ffbf7a] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:48:46.441 INFO 26087 --- [ocp-schedules-4,35901616bfd3a547,019dc8ee05f8f837] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:48:46.441 INFO 26087 --- [ocp-schedules-22,00ec383ce3dcd134,54aea11990f0285e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:48:46.441 INFO 26087 --- [ocp-schedules-4,35901616bfd3a547,019dc8ee05f8f837] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:46.441 INFO 26087 --- [ocp-schedules-22,00ec383ce3dcd134,54aea11990f0285e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:48:46.537 INFO 26087 --- [ocp-async-7,0b8a45bfc0b0abe7,1280bb2802b3adf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:48:46.539 INFO 26087 --- [ocp-schedules-20,6b5681f20287f09c,d04d916edbf58b36] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:48:46.599 INFO 26087 --- [ocp-schedules-3,288472404b77739e,f63bf52ca5000c99] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:48:46.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:48:46.623207+08:00, lastProcessTime=2025-04-08T21:48:36.623798+08:00 2025-04-08 21:48:46.629 INFO 26087 --- [ocp-schedules-15,2d83ad37181560c7,d5ffb79312cbb97d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:48:46.629 INFO 26087 --- [ocp-schedules-15,3bc8fd0104df3113,032307c20166d8f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:48:46.630 INFO 26087 --- [ocp-schedules-15,3bc8fd0104df3113,032307c20166d8f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:46.635 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:48:46.635 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:48:46.665 WARN 26087 --- [ocp-async-9,fb53923025111741,6e6e1c289454d88f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:46.665 INFO 26087 --- [ocp-async-10,a19cb1170bae6e92,fed4fdf58fbf293b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:46.736 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:48:46.623207+08:00 2025-04-08 21:48:46.809 INFO 26087 --- [ocp-schedules-6,790199ff1795a5ed,3c7e445a41b4a885] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:48:47.675 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:47.675 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:48.271 INFO 26087 --- [prometheus-minute-schedule-1,34602afda3576069,e0f9a305f8da4a04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:48.665 WARN 26087 --- [ocp-async-4,b9f49e2fc16ce117,e03ec001af5ee1cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:48.665 INFO 26087 --- [ocp-async-2,effca1fcf82355b4,2098ddce330c6139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:50.389 INFO 26087 --- [ocp-schedules-10,26c9ec397d0e1e59,762238f90026ed38] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:48:50.391 INFO 26087 --- [ocp-schedules-10,26c9ec397d0e1e59,762238f90026ed38] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:48:50.665 WARN 26087 --- [ocp-async-8,83ed75987e7ac120,32ed2283833f75c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:50.665 INFO 26087 --- [ocp-async-1,910390825f3383c8,ec285c0422f10887] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:50.675 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:50.676 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:51.271 INFO 26087 --- [prometheus-minute-schedule-1,67e0bcc19c40f2d2,18fdd914abef08c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:52.665 WARN 26087 --- [ocp-async-9,e5d7a6b719d5cf8b,d9cdaac8c29866f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:52.665 INFO 26087 --- [ocp-async-1,f72b1a22c0718d88,d3636863bd397294] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:53.676 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:53.677 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:53.695 INFO 26087 --- [ocp-schedules-28,ea07433a67a67789,c28b001c8a47559f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:48:54.271 INFO 26087 --- [prometheus-minute-schedule-1,5d42ac9e78c2448a,a77e15b3f350a7d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:54.665 WARN 26087 --- [ocp-async-10,24e910d4306177b7,7675954158aed5b6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:54.665 INFO 26087 --- [ocp-async-3,b5fe9d69650412fe,e967e161caa28ddf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:56.442 INFO 26087 --- [ocp-schedules-22,00ec383ce3dcd134,54aea11990f0285e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:48:56.442 INFO 26087 --- [ocp-schedules-4,35901616bfd3a547,019dc8ee05f8f837] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:48:56.514 INFO 26087 --- [ocp-schedules-27,fea6d0166bf17344,4a8d3f42c8097053] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-04-08 21:48:56.514 INFO 26087 --- [ocp-schedules-27,fea6d0166bf17344,4a8d3f42c8097053] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:48:56.515 INFO 26087 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-04-08 21:48:56.537 INFO 26087 --- [ocp-async-4,3eb452ae5746054d,9165fb285c802de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:48:56.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:48:56.623816+08:00, lastProcessTime=2025-04-08T21:48:46.623207+08:00 2025-04-08 21:48:56.665 WARN 26087 --- [ocp-async-5,b3e517953d06d14e,c43b3c108a3c5374] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:56.665 INFO 26087 --- [ocp-async-7,c337c060d23b3749,3b8459ae1a701168] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:56.677 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:56.677 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:48:56.747 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:48:56.623816+08:00 2025-04-08 21:48:56.810 INFO 26087 --- [ocp-schedules-4,f039817175010c79,31fe868845397b79] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:48:57.271 INFO 26087 --- [prometheus-minute-schedule-1,31b25facdf538f93,6b4982b73e1db693] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120140, interval=60 2025-04-08 21:48:58.567 INFO 26087 --- [ocp-schedules-23,00e2eb6ed34139c6,dc606984445fb5b5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:48:58.584 INFO 26087 --- [ocp-schedules-23,00e2eb6ed34139c6,dc606984445fb5b5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:48:58.584 INFO 26087 --- [ocp-schedules-23,00e2eb6ed34139c6,dc606984445fb5b5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:48:58.665 WARN 26087 --- [ocp-async-9,9a23506b57f0f5b9,1e42dd2b7d9df998] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:48:58.666 INFO 26087 --- [ocp-async-6,8d9de5d9b45a9397,50c4cd09d377be51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:48:59.678 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:48:59.678 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:00.271 INFO 26087 --- [prometheus-minute-schedule-1,930962627cd55f26,13364d3b64bfcfe9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:00.397 INFO 26087 --- [ocp-schedules-24,21e5c7e9615cd0e4,c464ff7d982eb133] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:49:00.398 INFO 26087 --- [ocp-schedules-24,21e5c7e9615cd0e4,c464ff7d982eb133] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:49:00.666 WARN 26087 --- [ocp-async-3,5d657c8f1928c42d,b8d6faa9ebe3637a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:00.666 INFO 26087 --- [ocp-async-10,9b0402de8dd6fcd5,5b68758d7bb1992b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:02.666 WARN 26087 --- [ocp-async-5,1ac7fccfb2e04995,70eb89336a2a0eaa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:02.666 INFO 26087 --- [ocp-async-10,8c94d87c761e253e,47a8f7a470df8871] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:02.678 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:02.679 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:03.271 INFO 26087 --- [prometheus-minute-schedule-1,ff3ce47e1bf94efc,bcc6303fb39c2f17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:03.716 INFO 26087 --- [ocp-schedules-21,1ca6a91297b8a0a1,2e74cfd8c59cac73] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:49:04.666 WARN 26087 --- [ocp-async-10,6468470325865e16,7b8cf5f07872b081] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:04.666 INFO 26087 --- [ocp-async-9,8d441b3d0faf0b4f,90c3c8ae645931ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:05.679 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:05.679 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:06.271 INFO 26087 --- [prometheus-minute-schedule-1,14864cb76dfc6a64,19255f27b0ef8021] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:06.272 INFO 26087 --- [prometheus-cache-monitor1,b2b779c42943edd3,a142f41fc697ff65] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":380,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:49:06.272 INFO 26087 --- [prometheus-cache-monitor1,b2b779c42943edd3,a142f41fc697ff65] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:49:06.496 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-04-08T21:43:06.451, end=2025-04-08T21:48:06.468} 2025-04-08 21:49:06.500 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [] 2025-04-08 21:49:06.500 INFO 26087 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:49:06.515 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-04-08T21:43:06.409, end=2025-04-08T21:48:06.471} 2025-04-08 21:49:06.515 INFO 26087 --- [ocp-schedules-27,fea6d0166bf17344,4a8d3f42c8097053] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:49:06.517 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [] 2025-04-08 21:49:06.517 INFO 26087 --- [pool-17-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-04-08 21:49:06.537 INFO 26087 --- [ocp-async-7,40c7f86d2fdb4e6a,17554b0c228337ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:49:06.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:49:06.624197+08:00, lastProcessTime=2025-04-08T21:48:56.623816+08:00 2025-04-08 21:49:06.666 WARN 26087 --- [ocp-async-6,ffe3fe0f73c0abf4,f95883ed202478f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:06.666 INFO 26087 --- [ocp-async-4,4ae47a78a2f5692c,835856d2db158fd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:06.736 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:49:06.624197+08:00 2025-04-08 21:49:06.812 INFO 26087 --- [ocp-schedules-7,528c6437303beb03,13c25573dde1d197] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:49:08.666 WARN 26087 --- [ocp-async-5,a368ec736d5a2e48,63869f667a355598] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:08.666 INFO 26087 --- [ocp-async-4,0877829d4b766809,d7feca913f7ad29a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:08.679 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:08.680 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:09.271 INFO 26087 --- [prometheus-minute-schedule-1,f719b414122254b4,5c9d6d7c0f1508ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:10.001 INFO 26087 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:49:10.001 INFO 26087 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:49:10.414 INFO 26087 --- [ocp-schedules-4,70a5393261900e98,aaa4f87acb859f99] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:49:10.415 INFO 26087 --- [ocp-schedules-4,70a5393261900e98,aaa4f87acb859f99] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:49:10.521 INFO 26087 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 4, ruleSize = 4. 2025-04-08 21:49:10.666 WARN 26087 --- [ocp-async-4,1d6d4fff3ab67cfe,23862b926064aec8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:10.666 INFO 26087 --- [ocp-async-7,49a2502095a19389,7be9e1aa9571f802] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:11.680 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:11.680 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:12.271 INFO 26087 --- [prometheus-minute-schedule-1,a18167bcd4a0f053,37df2f368fccc8dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:12.554 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1744120080, nodeCount=380, affectRows=380 2025-04-08 21:49:12.556 INFO 26087 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1744120080, nodeCount=377, validCount=377 2025-04-08 21:49:12.666 WARN 26087 --- [ocp-async-7,0b7eb9daed46bff4,3ccf3a48b976722c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:12.667 INFO 26087 --- [ocp-async-6,1966441e7c82fab9,4a5fda720bcedf0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:13.742 INFO 26087 --- [ocp-schedules-1,543dad4cc8f05a9b,6009382fa079fd87] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:49:14.667 WARN 26087 --- [ocp-async-5,9afefbd18c1d8f3c,7240d8ef70a16186] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:14.667 INFO 26087 --- [ocp-async-6,1e0281ba36800e56,51d8305a3e7b11c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:14.681 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:14.681 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:15.271 INFO 26087 --- [prometheus-minute-schedule-1,556253a52c9470f1,7523239b9b475e6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:16.433 INFO 26087 --- [ocp-schedules-22,0614e3a50d854006,89eb21bece1723e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-04-08 21:49:16.434 INFO 26087 --- [ocp-schedules-27,49c04cec9535fb14,c785af900c8071d3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-04-08 21:49:16.437 INFO 26087 --- [ocp-schedules-22,0614e3a50d854006,89eb21bece1723e8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-04-08 21:49:16.437 INFO 26087 --- [ocp-schedules-5,f517e5034b2f71ae,0131f9cf637c4e01] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:49:16.437 INFO 26087 --- [ocp-schedules-5,f517e5034b2f71ae,0131f9cf637c4e01] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:49:16.437 INFO 26087 --- [ocp-schedules-12,c964d56ed138a14a,91f155f4d131415d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:49:16.440 INFO 26087 --- [ocp-schedules-20,726516e49717391b,c4500712f37989b2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:49:16.440 INFO 26087 --- [ocp-schedules-20,726516e49717391b,c4500712f37989b2] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:49:16.440 INFO 26087 --- [ocp-schedules-26,f0bd82301e1c155e,ba215a1f360079c6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:49:16.440 INFO 26087 --- [ocp-schedules-26,f0bd82301e1c155e,ba215a1f360079c6] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:49:16.441 INFO 26087 --- [ocp-schedules-3,9a0eb3d7bd0b6ba8,2bb13b47917d2e0c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:49:16.441 INFO 26087 --- [ocp-schedules-5,921141dc80f70b29,a20940bd51483b44] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:49:16.441 INFO 26087 --- [ocp-schedules-13,527f32813e46d3b9,8e8a3f9526381107] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:49:16.441 INFO 26087 --- [ocp-schedules-5,921141dc80f70b29,a20940bd51483b44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:16.442 INFO 26087 --- [ocp-schedules-13,527f32813e46d3b9,8e8a3f9526381107] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:16.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:49:16.442 INFO 26087 --- [ocp-schedules-27,49c04cec9535fb14,c785af900c8071d3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-04-08 21:49:16.482 INFO 26087 --- [ocp-schedules-19,04d664fb3c5ac345,de315f476e0e9bfb] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[] 2025-04-08 21:49:16.482 INFO 26087 --- [ocp-schedules-19,04d664fb3c5ac345,de315f476e0e9bfb] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-04-08 21:49:16.486 INFO 26087 --- [ocp-schedules-28,b7d30a497e18c85c,4e3876292b88f15a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:49:16.486 INFO 26087 --- [ocp-schedules-28,b7d30a497e18c85c,4e3876292b88f15a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-04-08 21:49:16.506 INFO 26087 --- [ocp-schedules-7,96c4171cf73afc50,4fa1710e28fa4ea1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":1520,"minuteWriteQueue":0,"minuteReadCacheSize":0,"secondWriteCacheSize":378,"seriesIdKeySize":428,"secondWriteQueue":0,"minuteWriteCacheSize":379} 2025-04-08 21:49:16.506 INFO 26087 --- [ocp-schedules-7,96c4171cf73afc50,4fa1710e28fa4ea1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_cluster_status_check=9, system_cpu_count=2, jvm_threads_states_threads=8, jvm_memory_committed_bytes=10, jvm_gc_memory_promoted_bytes_total=2, ocp_meta_db_health=4, jvm_gc_pause_seconds_max=7, system_cpu_usage=2, ob_cluster_sync=8, cluster_cgroup_value=6, http_server_requests_seconds_count=64, jvm_gc_memory_allocated_bytes_total=2, jvm_threads_peak_threads=2, ocp_alarm_detect_duration_ms_count=134, jvm_memory_used_bytes=10, process_cpu_usage=2, jvm_memory_max_bytes=10, jvm_gc_max_data_size_bytes=2, http_server_requests_seconds_sum=64, jvm_gc_pause_seconds_sum=7, ocp_alarm_detect_duration_ms_sum=134, jvm_threads_live_threads=2, system_load_average_1m=2, jvm_gc_pause_seconds_count=7, jvm_gc_overhead_percent=2} 2025-04-08 21:49:16.538 INFO 26087 --- [ocp-async-4,b6c97ca0a47bb3da,58c3c54e0ca042f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:49:16.547 INFO 26087 --- [ocp-schedules-23,91fc0be0f94dffef,52a51009cecc39f9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-04-08 21:49:16.598 INFO 26087 --- [ocp-schedules-11,283a198241f6d30e,51a98f6659cbf198] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-04-08 21:49:16.598 INFO 26087 --- [ocp-schedules-11,283a198241f6d30e,51a98f6659cbf198] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-04-08 21:49:16.600 INFO 26087 --- [ocp-schedules-12,c812532a7220aa04,58c20c1bfa9872de] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:49:16.608 INFO 26087 --- [ocp-schedules-20,8e61179fec13860c,e2db1128fc930237] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-04-08 21:49:16.608 INFO 26087 --- [ocp-schedules-20,8e61179fec13860c,e2db1128fc930237] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-04-08 21:49:16.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:49:16.623053+08:00, lastProcessTime=2025-04-08T21:49:06.624197+08:00 2025-04-08 21:49:16.632 INFO 26087 --- [ocp-schedules-26,6fd96e9e15d9cbb4,ac7568e8b73d4991] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-04-08 21:49:16.633 INFO 26087 --- [ocp-schedules-26,6fd96e9e15d9cbb4,ac7568e8b73d4991] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-04-08 21:49:16.633 INFO 26087 --- [ocp-schedules-26,6fd96e9e15d9cbb4,ac7568e8b73d4991] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 1 2025-04-08 21:49:16.636 INFO 26087 --- [ocp-schedules-15,3bc8fd0104df3113,032307c20166d8f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:49:16.636 INFO 26087 --- [ocp-schedules-15,f55c661e5cec035f,09683b859bdd4f27] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:49:16.636 INFO 26087 --- [ocp-schedules-15,f55c661e5cec035f,09683b859bdd4f27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:16.641 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:49:16.641 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:49:16.658 INFO 26087 --- [ocp-schedules-3,94eccaa9c2d1f2fa,3d481f9e538e36e0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-04-08T21:49:16.658+08:00, clusters=[1000003] 2025-04-08 21:49:16.658 INFO 26087 --- [ocp-schedules-27,2bb29fedeaf43703,e7b1b814172b4499] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-04-08 21:49:16.658 INFO 26087 --- [ocp-schedules-3,94eccaa9c2d1f2fa,3d481f9e538e36e0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster ocpdb, trans stat result is empty. 2025-04-08 21:49:16.667 INFO 26087 --- [ocp-async-1,8ead0251e89c4dcd,7acf02528035b019] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:16.667 WARN 26087 --- [ocp-async-2,6675d153270a1711,17a40e7c1f36aed2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:16.672 INFO 26087 --- [ocp-schedules-28,3aaf0290e03fc6e8,c7a07c00ce9fa1fe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-04-08 21:49:16.672 INFO 26087 --- [ocp-schedules-28,3aaf0290e03fc6e8,c7a07c00ce9fa1fe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-04-08 21:49:16.674 INFO 26087 --- [ocp-schedules-25,7230a3923a74891b,5e02143e8382ec1d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-04-08T21:49:16.674+08:00, clusters=[1000003] 2025-04-08 21:49:16.676 INFO 26087 --- [ocp-schedules-25,7230a3923a74891b,5e02143e8382ec1d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster ocpdb for slowSqlStat, sql count is 0. 2025-04-08 21:49:16.677 INFO 26087 --- [ocp-schedules-28,3aaf0290e03fc6e8,c7a07c00ce9fa1fe] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=Collect all tenants major compactions, skip execute 2025-04-08 21:49:16.677 INFO 26087 --- [ocp-schedules-28,3aaf0290e03fc6e8,c7a07c00ce9fa1fe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-04-08 21:49:16.776 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:49:16.623053+08:00 2025-04-08 21:49:16.813 INFO 26087 --- [ocp-schedules-14,4cd33e560698c6fd,5a08b7d8cc48b197] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:49:16.841 INFO 26087 --- [ocp-schedules-21,2bed51a754383550,ee4ef2bd1dc69751] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-04-08 21:49:16.842 INFO 26087 --- [ocp-schedules-21,2bed51a754383550,ee4ef2bd1dc69751] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:16.853 ERROR 26087 --- [ocp-schedules-21,2bed51a754383550,ee4ef2bd1dc69751] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:47) at com.oceanbase.ocp.server.common.scheduled.ClusterSchedules.syncRootServer(ClusterSchedules.java:88) at sun.reflect.GeneratedMethodAccessor848.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=ocpdb,sys,ocp_monitor at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$b673b71f.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.buildConnectProperties(AbstractObOperatorFactory.java:324) at com.oceanbase.ocp.service.ob.obsdk.factory.AbstractObOperatorFactory.createObOperator(AbstractObOperatorFactory.java:117) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.getRsList(RefreshRsListTask.java:90) at com.oceanbase.ocp.service.task.business.cluster.RefreshRsListTask.processCluster(RefreshRsListTask.java:42) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:49:16.853 INFO 26087 --- [ocp-schedules-21,2bed51a754383550,ee4ef2bd1dc69751] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=1, successCount=0 2025-04-08 21:49:16.856 INFO 26087 --- [ocp-schedules-18,be79cf3fbb5e109e,101f3e06df40b3fa] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-04-08 21:49:16.856 INFO 26087 --- [ocp-schedules-18,be79cf3fbb5e109e,101f3e06df40b3fa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=0 2025-04-08 21:49:16.859 INFO 26087 --- [ocp-schedules-18,be79cf3fbb5e109e,101f3e06df40b3fa] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-04-08 21:49:16.859 INFO 26087 --- [ocp-schedules-18,be79cf3fbb5e109e,101f3e06df40b3fa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766209785803853824 2025-04-08 21:49:16.861 INFO 26087 --- [ocp-schedules-18,be79cf3fbb5e109e,101f3e06df40b3fa] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-04-08 21:49:16.861 INFO 26087 --- [ocp-schedules-18,be79cf3fbb5e109e,101f3e06df40b3fa] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=766208550866538496 2025-04-08 21:49:16.898 INFO 26087 --- [ocp-schedules-29,71ed93bb8ee8128f,cc76db5b253f801d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=1, successCount=1 2025-04-08 21:49:17.038 INFO 26087 --- [ocp-schedules-23,1f2c3e7831e96f2b,d8e0c22452ec57ad] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-04-08 21:49:17.039 INFO 26087 --- [ocp-schedules-23,1f2c3e7831e96f2b,d8e0c22452ec57ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:17.054 ERROR 26087 --- [Sync all tenant information1,,] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.NotFoundException occurred with code error.ob.tenant.name.not.found, and args [ocpdb, sys] 2025-04-08 21:49:17.055 ERROR 26087 --- [ocp-schedules-23,1f2c3e7831e96f2b,d8e0c22452ec57ad] .o.o.s.t.b.c.AbstractShardingClusterTask : process cluster error java.util.concurrent.ExecutionException: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.run(AbstractShardingClusterTask.java:77) at com.oceanbase.ocp.server.common.scheduled.TenantSchedules.syncAllTenantInfo(TenantSchedules.java:44) at sun.reflect.GeneratedMethodAccessor799.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.oceanbase.ocp.core.exception.NotFoundException: [OCP NotFoundException]: status=404 NOT_FOUND, errorCode=OB_TENANT_NAME_NOT_FOUND, args=ocpdb,sys at sun.reflect.GeneratedConstructorAccessor389.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.oceanbase.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.lambda$getSysTenant$5(TenantDaoManagerImpl.java:176) at java.util.Optional.orElseThrow(Optional.java:290) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl.getSysTenant(TenantDaoManagerImpl.java:176) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$FastClassBySpringCGLIB$$f93ff265.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.obops.internal.tenant.TenantDaoManagerImpl$$EnhancerBySpringCGLIB$$44d4f1d4.getSysTenant() at com.oceanbase.ocp.obops.internal.tenant.TenantSyncServiceImpl.markSysTenantUnavailable(TenantSyncServiceImpl.java:314) at com.oceanbase.ocp.service.task.business.tenant.SyncAllTenantInfoShardingTask.processCluster(SyncAllTenantInfoShardingTask.java:38) at com.oceanbase.ocp.service.task.business.cluster.AbstractShardingClusterTask.lambda$run$1(AbstractShardingClusterTask.java:70) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 common frames omitted 2025-04-08 21:49:17.055 INFO 26087 --- [ocp-schedules-23,1f2c3e7831e96f2b,d8e0c22452ec57ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=1, successCount=0 2025-04-08 21:49:17.157 INFO 26087 --- [ocp-schedules-1,a780579b9922e625,c739a5310e8c0c60] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-04-08 21:49:17.681 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:17.682 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:17.741 INFO 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[192.168.1.61] 2025-04-08 21:49:17.741 INFO 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000006 2025-04-08 21:49:17.743 INFO 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:49:17.744 WARN 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:49:17.744 INFO 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000006 2025-04-08 21:49:17.745 WARN 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.c.agent.HostAgentServiceImpl : OCP agent not found: hostId=1000006 2025-04-08 21:49:17.748 INFO 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://192.168.1.61:62889/api/v1/log/alarms, params:null 2025-04-08 21:49:17.750 WARN 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] c.o.o.s.c.scheduled.AbstractSchedules : Schedule task failed, methodName=getHostAgentAlarms, confName=getHostAgentAlarms 2025-04-08 21:49:17.751 ERROR 26087 --- [ocp-schedules-25,bd9ae1517d08fcca,b05b67e2e4f9eddd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:82) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:136) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpConnectFailedException: connect to agent failed, ConnectException: Connection refused (Connection refused) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:501) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doGet(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.get(HttpTemplate.java:85) at com.oceanbase.ocp.executor.executor.MonitorExecutor.getAlarms(MonitorExecutor.java:51) at com.oceanbase.ocp.compute.agent.manager.HostAgentAlarmManagerImpl.getHostAgentAlarms(HostAgentAlarmManagerImpl.java:62) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.getHostAgentAlarmsByHost(HostAgentServiceImpl.java:1106) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$55dbce47.getHostAgentAlarmsByHost() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$getHostAgentAlarms$1(HostSchedules.java:105) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650) at com.oceanbase.ocp.server.common.scheduled.HostSchedules.getHostAgentAlarms(HostSchedules.java:105) ... 14 common frames omitted Caused by: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:519) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:432) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:323) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.lambda$doGet$0(HttpTemplate.java:220) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.safeExecute(HttpTemplate.java:492) ... 40 common frames omitted Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 192.168.1.61:62889 [/192.168.1.61] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:471) ... 54 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 63 common frames omitted 2025-04-08 21:49:18.271 INFO 26087 --- [prometheus-minute-schedule-1,42e3186ea7fbab07,6b98668c79e39342] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:18.585 INFO 26087 --- [ocp-schedules-29,95dc4207be6308da,a3077b894af67b1c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:49:18.608 INFO 26087 --- [ocp-schedules-29,95dc4207be6308da,a3077b894af67b1c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:49:18.608 INFO 26087 --- [ocp-schedules-29,95dc4207be6308da,a3077b894af67b1c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:49:18.667 INFO 26087 --- [ocp-async-5,41170f516e6d5fac,2f44da856f2b96cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:18.667 WARN 26087 --- [ocp-async-8,ccb98a774b03a970,8c56577f088cd548] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:20.466 INFO 26087 --- [ocp-schedules-3,2db1588c8b6859c5,d9505ab92c68835a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:49:20.468 INFO 26087 --- [ocp-schedules-3,2db1588c8b6859c5,d9505ab92c68835a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:49:20.507 INFO 26087 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-04-08 21:49:20.667 WARN 26087 --- [ocp-async-9,1526a1353708c8c4,c60d9817f9bdca97] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:20.667 INFO 26087 --- [ocp-async-6,eb3c4c5e2c366515,8e786eece7eb19fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:20.682 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:20.682 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:21.271 INFO 26087 --- [prometheus-minute-schedule-1,59f928992e7128ba,d7da021a35d50c6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:22.667 WARN 26087 --- [ocp-async-2,f31790ec6ac97871,65d6e378da499e82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:22.667 INFO 26087 --- [ocp-async-4,4f8e1d9da14c327e,8ef95114a9be0885] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:23.682 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:23.683 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:23.762 INFO 26087 --- [ocp-schedules-26,61af77fde28393c7,7a0d01c3b643b4d2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:49:24.271 INFO 26087 --- [prometheus-minute-schedule-1,cf6175e9dc746ee6,3dcf6b979e22b5b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:24.667 WARN 26087 --- [ocp-async-4,6cc5a630befbe009,b74207042b2cd6aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:24.667 INFO 26087 --- [ocp-async-8,fb785a46665279d1,e9f785a098aa4cff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:26.442 INFO 26087 --- [ocp-schedules-5,921141dc80f70b29,a20940bd51483b44] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:49:26.442 INFO 26087 --- [ocp-schedules-13,527f32813e46d3b9,8e8a3f9526381107] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:49:26.538 INFO 26087 --- [ocp-async-5,e4b942dab3039c97,4a3a9186a31a418e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:49:26.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:49:26.622728+08:00, lastProcessTime=2025-04-08T21:49:16.623053+08:00 2025-04-08 21:49:26.668 WARN 26087 --- [ocp-async-2,614b7735a909de39,6abc009fb8e9717b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:26.668 INFO 26087 --- [ocp-async-10,351efce8e559aa43,c5b88c3ba1ed4c3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:26.683 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:26.683 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:26.756 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:49:26.622728+08:00 2025-04-08 21:49:26.815 INFO 26087 --- [ocp-schedules-5,6e57310cb0d46d52,4503687b884d588c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:49:27.271 INFO 26087 --- [prometheus-minute-schedule-1,64a9019921a6334c,df70cd52a8965eda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:28.668 WARN 26087 --- [ocp-async-4,7c84141629859578,21c64bace84eb413] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:28.668 INFO 26087 --- [ocp-async-10,3621b34e29e2635e,7efcbfa5e373b1cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:29.684 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:29.684 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:30.271 INFO 26087 --- [prometheus-minute-schedule-1,d6c8f252f0ce49de,15fc6f8140e0b614] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:30.471 INFO 26087 --- [ocp-schedules-4,02a485b4b4d5f921,ce170370b3a26504] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:49:30.472 INFO 26087 --- [ocp-schedules-4,02a485b4b4d5f921,ce170370b3a26504] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:49:30.668 WARN 26087 --- [ocp-async-5,8928480c7e260780,a85a8e38b5de3ddb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:30.668 INFO 26087 --- [ocp-async-1,6b4e956664192725,1280a339ec2a9400] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:32.668 WARN 26087 --- [ocp-async-2,0ab9878155e3feb1,7ab7e4c3238d79a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:32.668 INFO 26087 --- [ocp-async-6,a466d3148f8b6f1b,526e59d53d3f7447] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:32.684 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:32.685 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:33.271 INFO 26087 --- [prometheus-minute-schedule-1,c2ceb598a552a9f9,01cdd47d02bff099] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:33.787 INFO 26087 --- [ocp-schedules-19,8eda5456f19ecbfa,826059683cd37473] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:49:34.668 WARN 26087 --- [ocp-async-4,c20e8b290b68fb65,86afd6c128638079] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:34.668 INFO 26087 --- [ocp-async-7,b13e15a9e9ba8d67,6b549697dfbc89e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:34.910 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:49:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:49:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-04-08 21:49:34.911 INFO 26087 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-04-08 21:49:34.912 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-04-08 21:49:34.916 WARN 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Skip cluster ocpdb(1000003), status=TAKINGOVER 2025-04-08 21:49:34.916 INFO 26087 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=4ms, period=120000ms 2025-04-08 21:49:35.685 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:35.686 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:36.271 INFO 26087 --- [prometheus-minute-schedule-1,c32bbdab6a0ba55f,42fd88857f15ff2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:36.538 INFO 26087 --- [ocp-async-7,5d4818e18a6ec9fd,2f7e1375e9708df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:49:36.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:49:36.623659+08:00, lastProcessTime=2025-04-08T21:49:26.622728+08:00 2025-04-08 21:49:36.668 INFO 26087 --- [ocp-async-5,64c31622158a9233,48063893c1549aa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:36.668 WARN 26087 --- [ocp-async-1,cdf2b49e2d125d53,aa92e7193687657e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:36.708 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:49:36.623659+08:00 2025-04-08 21:49:36.816 INFO 26087 --- [ocp-schedules-6,3dd3b8312754ae48,8f873e0ae6bd8942] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:49:38.609 INFO 26087 --- [ocp-schedules-18,4c8ec2641df7f11e,54635ceaf48df661] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-04-08 21:49:38.615 INFO 26087 --- [ocp-schedules-18,4c8ec2641df7f11e,54635ceaf48df661] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:49:38.615 INFO 26087 --- [ocp-schedules-18,4c8ec2641df7f11e,54635ceaf48df661] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[254bed2a-3c53-4ecf-b850-9c2422739b3a] 2025-04-08 21:49:38.668 WARN 26087 --- [ocp-async-9,56a53dafbd5de836,2c0422ecd1988806] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:38.668 INFO 26087 --- [ocp-async-6,8527f69fe1901bab,945d995c791126d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:38.686 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:38.686 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:39.271 INFO 26087 --- [prometheus-minute-schedule-1,c9b84571824a2c31,b5e85dc6fd945b24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:40.002 INFO 26087 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=0 2025-04-08 21:49:40.002 INFO 26087 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=0 2025-04-08 21:49:40.476 INFO 26087 --- [ocp-schedules-10,4d3b788f6f0b5f03,d7435564d899dc9d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:49:40.477 INFO 26087 --- [ocp-schedules-10,4d3b788f6f0b5f03,d7435564d899dc9d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:49:40.669 WARN 26087 --- [ocp-async-3,134f494f95532286,5987ede9d823d7ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:40.669 INFO 26087 --- [ocp-async-7,b3c0cafb3ecd6f01,19dca2ae81f2b8c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:41.686 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:41.687 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:42.271 INFO 26087 --- [prometheus-minute-schedule-1,32f35db4ee80b07d,9de7124072db50cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:42.669 WARN 26087 --- [ocp-async-8,60751cc61fcefbb6,8af55d04ec4de50c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:42.669 INFO 26087 --- [ocp-async-1,b4beb506bd122bb7,ad3141371172afaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:43.809 INFO 26087 --- [ocp-schedules-16,e8b72cce18181703,ca8c23f6f07add47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:49:44.669 WARN 26087 --- [ocp-async-9,ce12ea294a138101,9b714df734a7a40b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:44.669 INFO 26087 --- [ocp-async-5,9dead8b8bae2669c,7ab3ff32806935ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:44.687 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:44.688 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:45.271 INFO 26087 --- [prometheus-minute-schedule-1,06540017bd477dd0,a1c4123a2ddb5f06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:46.437 INFO 26087 --- [ocp-schedules-16,4587ad9c9281f526,e65f086c8f84fb7a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-04-08 21:49:46.438 INFO 26087 --- [ocp-schedules-5,e6e7e98884fe91fb,66def3dc3cbf49d7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[] 2025-04-08 21:49:46.438 INFO 26087 --- [ocp-schedules-5,e6e7e98884fe91fb,66def3dc3cbf49d7] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-status, skip execute 2025-04-08 21:49:46.440 INFO 26087 --- [ocp-schedules-6,40223970ce55ab2b,37ed6e7bb9edea5d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[] 2025-04-08 21:49:46.440 INFO 26087 --- [ocp-schedules-6,40223970ce55ab2b,37ed6e7bb9edea5d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-host-status, skip execute 2025-04-08 21:49:46.440 INFO 26087 --- [ocp-schedules-19,053f6dc66f534b1a,75a987784744e534] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[] 2025-04-08 21:49:46.440 INFO 26087 --- [ocp-schedules-19,053f6dc66f534b1a,75a987784744e534] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-clockdiff, skip execute 2025-04-08 21:49:46.441 INFO 26087 --- [ocp-schedules-2,613d26d8fa681cd4,e0bf022a24c12471] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-04-08 21:49:46.441 INFO 26087 --- [ocp-schedules-14,fec39ec147deb5ed,103e9b04194d94f4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-04-08 21:49:46.441 INFO 26087 --- [ocp-schedules-14,fec39ec147deb5ed,103e9b04194d94f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:46.441 INFO 26087 --- [ocp-schedules-2,613d26d8fa681cd4,e0bf022a24c12471] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:46.442 INFO 26087 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:49:46.481 INFO 26087 --- [ocp-schedules-10,80764e5bb7e9a997,cf3fd32b52c92e49] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-04-08 21:49:46.538 INFO 26087 --- [ocp-async-2,3ea8438a9cf0d486,e7ed1551fcf47101] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:49:46.601 INFO 26087 --- [ocp-schedules-29,6a1cdb9fd2714e4c,dee5f612581985e1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-04-08 21:49:46.624 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:49:46.623603+08:00, lastProcessTime=2025-04-08T21:49:36.623659+08:00 2025-04-08 21:49:46.642 INFO 26087 --- [ocp-schedules-15,f55c661e5cec035f,09683b859bdd4f27] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:49:46.642 INFO 26087 --- [ocp-schedules-15,9e62cd53384b1d5e,5f8961e24a781610] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-04-08 21:49:46.642 INFO 26087 --- [ocp-schedules-15,9e62cd53384b1d5e,5f8961e24a781610] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1000003] 2025-04-08 21:49:46.647 INFO 26087 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is not accessible, skip sync cluster, clusterId=1000003, status=TAKINGOVER 2025-04-08 21:49:46.648 INFO 26087 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=1000003, name=ocpdb, obClusterId=1744114127, vpcId=1, vpcName=system-default, type=PRIMARY, status=TAKINGOVER, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson=null, createTime=2025-04-08T21:34:15+08:00, creator=admin, obVersion=4.2.1.11, packageOperatingSystem=null, loadType=null, tenantCount=0) 2025-04-08 21:49:46.669 INFO 26087 --- [ocp-async-3,da8d5af08a3b8fa0,d565effe13786a8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:46.669 WARN 26087 --- [ocp-async-4,eece412979fb4880,17bd05432351b4fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:46.715 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:49:46.623603+08:00 2025-04-08 21:49:46.818 INFO 26087 --- [ocp-schedules-22,d7224ab8ddc30000,d80243c35bad8872] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:49:47.688 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:47.689 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:48.271 INFO 26087 --- [prometheus-minute-schedule-1,d127f56b59ccb04a,9d20886f37cd0e27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:48.669 WARN 26087 --- [ocp-async-1,fc2b292c58bf3805,ed47d078bcbcd02a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:48.669 INFO 26087 --- [ocp-async-7,8abcb67f8316339d,ff5bb81b17801b13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:50.504 INFO 26087 --- [ocp-schedules-19,2d86441c3d2c577e,b4ba84b37a2322c9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-04-08 21:49:50.505 INFO 26087 --- [ocp-schedules-19,2d86441c3d2c577e,b4ba84b37a2322c9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-04-08 21:49:50.669 WARN 26087 --- [ocp-async-2,9f491c0988a0abd8,71fc77a446b888af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:50.669 INFO 26087 --- [ocp-async-5,5d0d98272883e791,73197751f9fc0232] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:50.689 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:50.689 INFO 26087 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:51.271 INFO 26087 --- [prometheus-minute-schedule-1,5de4e5717708f844,6619e5f2cb662456] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:52.669 WARN 26087 --- [ocp-async-10,e15d153a7d1eb83d,5c1459ff793838f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:52.670 INFO 26087 --- [ocp-async-6,b22d6be33e47e537,37cf06d47724de0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:53.689 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:53.690 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:53.838 INFO 26087 --- [ocp-schedules-16,1bfe873a503e3476,93a9b8b1b3d25130] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-04-08 21:49:54.271 INFO 26087 --- [prometheus-minute-schedule-1,08be803ca4246a2d,047926d2a890ab18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60 2025-04-08 21:49:54.670 WARN 26087 --- [ocp-async-3,d5ec1c7d30206f00,2413b1543fc607ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:54.670 INFO 26087 --- [ocp-async-1,ef0ca4d4421401a6,55d190916aee4b50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:56.442 INFO 26087 --- [ocp-schedules-14,fec39ec147deb5ed,103e9b04194d94f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=1, successCount=1 2025-04-08 21:49:56.442 INFO 26087 --- [ocp-schedules-2,613d26d8fa681cd4,e0bf022a24c12471] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=1, successCount=1 2025-04-08 21:49:56.538 INFO 26087 --- [ocp-async-8,3feafd30be84a268,768e787890fe98b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-04-08 21:49:56.623 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-04-08T21:49:56.622956+08:00, lastProcessTime=2025-04-08T21:49:46.623603+08:00 2025-04-08 21:49:56.670 WARN 26087 --- [ocp-async-2,34f3256ad9cf7df6,27caa869f9d75a70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-04-08 21:49:56.670 INFO 26087 --- [ocp-async-9,6439a700cf161f0e,d9a9a62e3f526450] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-04-08 21:49:56.690 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 192.168.1.61:8080 refreshes policies... 2025-04-08 21:49:56.690 INFO 26087 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-04-08 21:49:56.748 INFO 26087 --- [alarm-task-1,8c622a7b8e880a84,864c7beb1db6dfab] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-04-08T21:49:56.622956+08:00 2025-04-08 21:49:56.819 INFO 26087 --- [ocp-schedules-8,075d0cb63fff7c3e,131f07f4e9e0cb35] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-04-08 21:49:57.271 INFO 26087 --- [prometheus-minute-schedule-1,2c7c0c1a38fb9c48,083ad397c2db11ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1744120200, interval=60